From d84c88f830b362221b0f3cfc1823602ef7c6c356 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Tue, 23 Jun 2026 21:50:58 -0400 Subject: [PATCH] docs: module map, RFC index, refactor plan (arch-refactor tier 0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Navigability groundwork from the architecture review — zero behaviour change. - docs/MODULES.md: the repo map. Area -> directory -> key files -> re-mint?, plus per-feature touch points (tree-shaking, direct-linking, numeric fl/fx, inlining, multimethods, deps) and where a given clojure.core fn lives. Answers "where does X live / what's related to Y" in one read. - docs/rfc/README.md: index the 7 RFCs; flags RFC 0007's stale "no code yet" status (direct-linking + tree-shaking shipped) and the undocumented inlining/numeric work. - CLAUDE.md: document the var-deref calling convention (public defns reached from the .ss runtime by string lookup aren't dead), the def-var! native pattern, and the overlay shadowing rule; point at MODULES.md. - REFACTOR_PLAN.md: the prioritized, risk-tiered plan (working doc for this branch). --- CLAUDE.md | 12 +++++ REFACTOR_PLAN.md | 117 +++++++++++++++++++++++++++++++++++++++++++++ docs/MODULES.md | 92 +++++++++++++++++++++++++++++++++++ docs/rfc/README.md | 21 ++++++++ 4 files changed, 242 insertions(+) create mode 100644 REFACTOR_PLAN.md create mode 100644 docs/MODULES.md create mode 100644 docs/rfc/README.md diff --git a/CLAUDE.md b/CLAUDE.md index 56b1834..1e0955d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,6 +109,18 @@ Issue tracking and design notes live in beads (`bd prime`, `bd memories`). forever. Use `jolt.host/ref-get`. - **Map literals with `:jolt/type` as a key** parse as tagged reader forms — don't tag overlay value maps in source. +- **The compiler is reached from the runtime by `var-deref` string lookup.** The + `.ss` runtime calls into the cross-compiled compiler with + `(var-deref "jolt.analyzer" "analyze")` etc., and the compiler resolves its own + unqualified `jolt.host/…` refs the same way against `host-contract.ss`. So a + public `defn` with no in-Clojure callers may still be a live entry point — don't + treat it as dead. Only a private `defn-` with no callers is safe to remove. +- **A native `clojure.core` fn is a `(def-var! "clojure.core" "name" …)`** in a + `host/chez/*.ss`; the rest of core is the overlay (`jolt-core/clojure/core/*.clj`). + A few natives are re-asserted in `post-prelude.ss` so they win over the overlay. + See [docs/MODULES.md](docs/MODULES.md) for where a given fn lives, and + [docs/seed-overlay-registry.md](docs/seed-overlay-registry.md) for the shadowing + rule. Start at [docs/MODULES.md](docs/MODULES.md) to find a feature's files. - **Fix latent bugs to match Clojure** rather than preserving them, with a regression case. Match the JVM (or provide a superset); the JVM-sourced corpus is the contract. diff --git a/REFACTOR_PLAN.md b/REFACTOR_PLAN.md new file mode 100644 index 0000000..6c0c50f --- /dev/null +++ b/REFACTOR_PLAN.md @@ -0,0 +1,117 @@ +# Architecture refactor plan + +Working doc for the `spike/arch-refactor` branch. Goal: make the codebase easier +to understand and safely modify — optimized for both human and LLM maintainers. +Derived from a four-part review (compiler core, Chez runtime, build/tooling, +top-level organization) plus call-graph analysis of the compiler. + +The codebase is in better shape than its 58-file runtime suggests: `rt.ss` is a +real load-order map, the `passes/` split is clean, filename prefixes encode tier +order, and the Makefile self-documents the gate. The work below targets the few +genuine liabilities. + +## The re-mint constraint shapes risk + +- **Seed sources** (`jolt-core/jolt/*.clj`, `jolt-core/clojure/core/*.clj`, + `host/chez/reader.ss`): a change needs `make remint` + the byte-fixpoint + `make selfhost`. Behaviour-preserving refactors are *verified* by selfhost, but + every touch costs a re-mint. Higher risk, batch them. +- **Runtime `.ss` + runtime-loaded `.clj`** (`build.ss`, `emit-image.ss`, + `main.clj`, the dispatcher shims, …): no re-mint; verified by `make test` + + `make shakesmoke`. Lower risk. +- **Docs / renames of non-seed files**: zero behaviour risk. + +## Tier 0 — navigability (zero risk, highest LLM value, do first) + +1. **`docs/MODULES.md` — the repo map.** One table: area → directory → key files → + re-mint? Plus a per-feature "touch points" list for cross-cutting features + (tree-shaking, direct-linking, numeric fl/fx, multimethods, the deps resolver). + Collapses the current 3–4-source lookup ("where does X live?") into one read. +2. **`docs/rfc/README.md`** — index the 7 RFCs (number, title, status), mirroring + `docs/spec/README.md`. +3. **Document the shipped-but-undocumented compiler features**: IR inlining and + fl*/fx* numeric lowering (the `passes/` pipeline). A short "compilation passes" + doc or an RFC, referenced from RFC 0007. +4. **CLAUDE.md — one "Invariants you must preserve" block**: the `var-deref` + calling convention (the compiler is reached from `.ss` by string lookup), the + `def-var!` native-binding pattern, and the seed dual-home shadowing rule (today + only in `seed-overlay-registry.md`). +5. **First-line purpose comment on every `.ss`** (audit the laggards) and note the + reader's surprising location (`host/chez/reader.ss`, Scheme, re-mint applies). + +## Tier 1 — runtime structure (no re-mint; verify with `make test` + shakesmoke) + +6. **Registry pattern for the core dispatchers — the biggest runtime win.** + `jolt-pr-str`, `jolt-pr-readable`, `jolt=2`, `jolt-get`, `jolt-class`, + `jolt-hash` are extended by `set!`-capture-and-rebind across ~8 files (e.g. + `jolt-pr-str` is wrapped 9× in 8 files). The registry pattern already exists in + tree (`register-str-render!`, `register-instance-check-arm!`) and is strictly + better: type-disjoint arms gathered in one walker, load-order-independent, + greppable. Add `register-pr-arm!` / `register-eq-arm!` / `register-get-arm!` / + `register-hash-arm!` / `register-class-arm!`; convert the ~35 `set!` sites to + one-line registrations. Eliminates "read 8 files to understand one dispatcher." +7. **Extract a `host/chez/dce.ss` module for tree-shaking.** The DCE logic is split + across `emit-image.ss` (the `dce-*` helpers + record producer) and `build.ss` + (`bld-shake-all` + the manifest splice). Pull every `dce-*` def + `bld-shake-all` + (→ `dce-shake`) into `dce.ss`; give the record a named accessor API + (`dce-rec-keep?`/`-fqn`/`-refs`/`-str`) instead of `(vector-ref r 0..3)`; split + `bld-shake-all`'s five jobs (root-seed, edge-build, BFS, bail-detect, partition) + into named steps. This is recently-added code and the loosest contract in the + build. +8. **Tag the runtime manifest.** `bld-emit-runtime` decides "splice shaken core" / + "drop compiler" by substring-matching `(load "…seed/prelude.ss")` strings. + Replace the manifest with tagged entries (`(prelude)`/`(image)`/`(compile-eval)`/ + `(load path)`) and dispatch on the tag — removes a silent-failure coupling. +9. **`build-binary` options map.** It takes 8 positional args ending in two bare + booleans (`direct-link?` `tree-shake?`); a swap compiles and misbehaves silently. + Pass one options map from `main.clj`; new flags become additive. +10. **`dynamic-wind` the compiler-global set/reset** in `build-binary` so + `set-optimize!`/`set-direct-link!` always revert (today they leak on a build + error; harmless for the CLI, a trap for any in-process caller). +11. **Re-split `host-static-{,statics,objects}.ss`** on a real axis — static + methods/fields vs host object classes — instead of the current "grew too big" + chain (one file is 502 lines, the largest non-seed runtime file). Move the + System-property/env plumbing out to `io.ss` / a `host-system.ss`. +12. **Consolidate the seq/transducer/parity natives** and dissolve + `natives-parity.ss` (a literal leftovers file). One `natives-transduce.ss` for + `td-*`/`into-xform`/`transduce`/`sequence`/`cat`; relocate parity's hash / + macroexpand / reader-conditional pieces to their real homes. +13. **Smaller renames/moves:** `dynamic-vars.ss` (5 constants) vs `dyn-binding.ss` + (the binding stack) — rename the former; pull the `format` engine out of + `natives-misc.ss` into `natives-format.ss`. + +## Tier 2 — compiler core (seed sources; re-mint + selfhost + corpus) + +14. **Split the success-type checker out of `types.clj`** (716 lines = inference + + checker + driver). Move the checker (`check-*`, the error-domain predicates) to + `jolt.passes.types.check`. Highest reduction in edit blast-radius in the + compiler — inference and checker stop being able to break each other. +15. **Decompose `infer`'s `:invoke` arm** (≈120 lines, 8 hand-coded call patterns) + into named `infer-` helpers (as `numeric.clj`'s `an-invoke` already + does), and add `ty`/`nd` accessors for the positional `[type node]` tuple + (a transposition is currently silent and type-correct). +16. **Single-source the const-keyword-lookup recognizer.** It's implemented three + times divergently (`inline.clj`, `types.clj` ×2 arms, `backend_scheme.clj`). + Lift one predicate into `jolt.passes.fold` (the shared-predicate home) for the + two analysis copies; the backend's value-emission copy can stay. +17. **Co-locate the numeric op tables.** `backend_scheme.clj` (`dbl-ops`/`lng-ops`, + the Scheme strings) and `numeric.clj` (`dbl-spec`/`lng-spec`, specializable + names) must agree or `emit-numeric` splices a `nil` op string. Cross-link or + share the name→op table. +18. **Collapse `local-escapes?`'s mechanical recursion** (≈12 of its arms are just + "does any child escape" = `reduce-ir-children`); keep only the binder/lookup + arms explicit. Removes ~50 lines and a new-op soundness hazard. Fold + `recur-kinds`/`recur-arg-lists` (numeric) into one `recur-tails`. +19. **Touch-when-nearby:** drop the dead `form-char?` refer in `analyzer.clj:21`; + standardize on `(get node :k)` in the pass files; promote a few inline-comment + soundness arguments to function docstrings (`an-invoke` `:wild`, `dbl/lng-spec`). + +## Sequencing + +Tier 0 lands immediately (docs only). Tier 1 is the bulk of the maintainability win +and carries no re-mint — do 6 (registry) and 7–10 (DCE module + build hygiene) +first; they target the code most likely to be edited and most recently grown. +Tier 2 batches into one or two re-mints once Tier 1 is stable, each verified by +`make selfhost` + `make corpus`. + +Every code change in Tiers 1–2 is behaviour-preserving and gated; no feature work. diff --git a/docs/MODULES.md b/docs/MODULES.md new file mode 100644 index 0000000..e95fa00 --- /dev/null +++ b/docs/MODULES.md @@ -0,0 +1,92 @@ +# Module map + +Where things live and what to read before changing them. Start here to answer +"where does feature X live?" and "what else do I need to touch?" + +## Areas + +| Area | Directory | Responsibility | Re-mint? | +| --- | --- | --- | --- | +| Chez runtime | `host/chez/*.ss` | The substrate: value model, persistent collections, seqs, vars/namespaces, host interop, native `clojure.core` shims, regex, FFI, IO, the **reader**. Composed by `rt.ss`. | only `reader.ss` | +| Compiler | `jolt-core/jolt/*.clj` | analyzer → IR → backend, the optimization passes, the CLI, the deps resolver, nREPL. Baked into the seed. | **yes** | +| `clojure.core` overlay | `jolt-core/clojure/core/NN-*.clj` | Portable `clojure.core` in dependency-ordered tiers (`00-syntax` … `50-io`); the `NN` prefix *is* the load order. | **yes** | +| Stdlib | `stdlib/clojure/*.clj` | Lazily-loaded portable namespaces (string/set/walk/edn/pprint/zip/test/data). | no | +| Build & tooling | `host/chez/build.ss`, `emit-image.ss`, `compile-eval.ss`, `loader.ss`, `cli.ss`, `bootstrap.ss` | AOT binary build, cross-compile, runtime eval/load, CLI spine, seed mint. | no (except via `reader.ss`) | +| Tests & gate | `test/chez/`, `test/conformance/`, `host/chez/run-*.ss`, `Makefile` | Corpus (JVM oracle), unit, per-feature tests. Every `make` target has a comment. | no | + +**The reader is in `host/chez/reader.ss`** (Scheme, a seed source) — *not* in +`jolt-core/jolt/` with the rest of the compiler. Re-mint applies to it. + +`rt.ss` is the runtime's load-order manifest: it `(load …)`s every shim in +dependency order with a per-file comment. Read it to see how the runtime is +composed and where a given `.ss` fits. + +## `host/chez/*.ss` by family + +- **Value model**: `values.ss` (nil/numbers/keywords/symbols), `collections.ss` + (persistent vec + HAMT map/set), `seq.ss` + `lazy-bridge.ss` (seqs, lazy-seqs), + `transients.ss`, `records.ss` + `records-interop.ss`. +- **Native `clojure.core` shims**: `natives-*.ss` (array/coll/meta/misc/num/parity/ + queue/seq/str/xform), plus `predicates.ss`, `converters.ss`, `printing.ss`. +- **Vars / namespaces / dynamics**: `vars.ss`, `ns.ss`, `dyn-binding.ss` (the + thread-local binding stack), `dynamic-vars.ss` (a few `*…*` constant defaults), + `atoms.ss`, `multimethods.ss`. +- **Host interop**: `host-class.ss` (class tokens + method dispatch), + `host-static*.ss` (static methods/fields + host object classes), `host-table.ss`, + `host-contract.ss` (the `jolt.host` seam the compiler resolves against), + `dot-forms.ss`, `records-interop.ss`. +- **Scalars / misc**: `regex.ss` (vendored irregex), `math.ss`, `inst-time.ss`, + `bigdec.ss`, `syntax-quote.ss`. +- **IO / system / concurrency / FFI**: `io.ss`, `png.ss`, `concurrency.ss`, + `async.ss`, `ffi.ss`. +- **Compiler entry on Chez**: `reader.ss`, `compile-eval.ss`, `emit-image.ss`, + `loader.ss`, `cli.ss`, `build.ss`, `bootstrap.ss`. + +## Where is a `clojure.core` fn implemented? + +Two homes, with a defined precedence: + +1. **Native shim** — a `(def-var! "clojure.core" "name" …)` in a `host/chez/*.ss` + (hot/representation-coupled fns: `first`, `get`, `=`, the predicates). +2. **Overlay** — a `defn` in a `jolt-core/clojure/core/NN-*.clj` tier (most of + `clojure.core`, in portable Clojure). +3. **`post-prelude.ss`** re-asserts a handful of natives *after* the overlay loads, + so the native version wins (the overlay's value-reading versions are wrong for + Chez-native chars/atoms/etc.). Each entry there says why. + +`grep 'def-var! "clojure.core" "frequencies"' host/chez` and +`grep -rn 'defn frequencies' jolt-core/clojure/core` to find a given fn. See +[seed-overlay-registry.md](seed-overlay-registry.md) for the shadowing mechanism. + +## Cross-cutting features — touch points + +A feature's *core* lives in one file; these are the other files you must keep in +sync when changing it. + +- **Tree-shaking / DCE** (`--tree-shake`): `emit-image.ss` (the `dce-*` helpers + + record producers) and `build.ss` (`bld-shake-all` reachability + the manifest + splice in `bld-emit-runtime`); the flag in `main.clj`; validated by + `host/chez/tree-shake-smoke.sh` (`make shakesmoke`) and `build-smoke.sh`. See + [tools-deps.md](tools-deps.md#tree-shaking). +- **Direct-linking** (`--direct-link`): `backend_scheme.clj` (`direct-link?`, + `emit-top-form`, the `jv$` bindings); `build.ss` turns it on; `main.clj` the + flag; `test/chez/directlink-test.ss`. +- **Numeric fl*/fx\*** (`^double`/`^long` hints): `jolt-core/jolt/passes/numeric.clj` + (the hint-directed pass + loop-counter + `:coerce`); `backend_scheme.clj` + (`dbl-ops`/`lng-ops` op strings, `emit-numeric`, entry/return coercion); + `analyzer.clj` (`nhint-of`, `:nhints`, `with-ret-nhint`); `host-contract.ss` + (`:num-ret` on resolve); `rt.ss` (`jolt->fx`); `test/chez/numeric-test.ss`. +- **IR inlining** (under `--opt`): `passes/inline.clj` (splice) + `passes.clj` + (stash) + `host-contract.ss` (`inline-ir`/`stash-inline!`); `test/chez/inline-test.ss`. +- **Multimethods**: `host/chez/multimethods.ss` (dispatch) + the overlay + `defmulti`/`defmethod` macros + `host-contract.ss` late-bind. +- **Deps resolution**: `jolt-core/jolt/deps.clj` (the only file) + `main.clj` + (applies the roots) + `loader.ss` (the `require` path). + +## Conventions you must preserve + +See **CLAUDE.md → "Conventions & Patterns"** for the load-bearing rules: the +re-mint trigger, the tier macro-ordering rule, the `get`-on-your-own-wrapper trap, +`:jolt/type`-as-a-key parsing, the `var-deref` calling convention (the compiler is +reached from the `.ss` runtime by string lookup, so a public `defn` with no +in-Clojure callers can still be live), and the writing style. diff --git a/docs/rfc/README.md b/docs/rfc/README.md new file mode 100644 index 0000000..083d9c3 --- /dev/null +++ b/docs/rfc/README.md @@ -0,0 +1,21 @@ +# RFCs + +Design notes for non-obvious language and compiler decisions. An RFC records *why* +a thing is built the way it is; the code is the source of truth for *how*. + +| # | Title | Status | Governs | +| --- | --- | --- | --- | +| [0001](0001-language-specification.md) | A Specification for the Clojure Language | Draft | The conformance target — what "is Clojure" means for jolt. | +| [0002](0002-reader-conditional-features.md) | Reader-Conditional Feature Set | Accepted | `#?(...)` feature keys (`:jolt`, `:clj`, `:default`). | +| [0003](0003-transients.md) | Transients | Accepted | `transient`/`persistent!` semantics + the Chez mutable backing. | +| [0004](0004-type-hints.md) | Type hints + keyword-lookup specialization | Accepted | `^Type`/`^:struct` hints → the bare-`get` fast path. | +| [0005](0005-structural-type-inference.md) | Structural collection-type inference | Implemented | The `:struct`/`:vec`/`:set` lattice in `passes/types`. | +| [0006](0006-success-type-checking.md) | Success typing (provably-wrong-code detection) | Implemented | The error-domain checker in `passes/types`. | +| [0007](0007-compilation-modes-and-binary-output.md) | Compilation modes + binary output | Implemented (doc lags) | `release`/`--opt`/`--dev`, `--direct-link`, `--tree-shake`. | + +RFC 0007's own status line still says "Draft, no code yet" — that is stale: +direct-linking and tree-shaking shipped (see [tools-deps.md](../tools-deps.md) and +`backend_scheme.clj` / `build.ss`). Two compiler features that grew alongside it — +**IR inlining** (`passes/inline.clj`, under `--opt`) and **numeric `fl*`/`fx*` +lowering** from `^double`/`^long` hints (`passes/numeric.clj`) — are not yet written +up as RFCs; their touch points are in [../MODULES.md](../MODULES.md).