docs: module map, RFC index, refactor plan (arch-refactor tier 0)

Navigability groundwork from the architecture review — zero behaviour change.

- docs/MODULES.md: the repo map. Area -> directory -> key files -> re-mint?, plus
  per-feature touch points (tree-shaking, direct-linking, numeric fl/fx, inlining,
  multimethods, deps) and where a given clojure.core fn lives. Answers "where does
  X live / what's related to Y" in one read.
- docs/rfc/README.md: index the 7 RFCs; flags RFC 0007's stale "no code yet" status
  (direct-linking + tree-shaking shipped) and the undocumented inlining/numeric work.
- CLAUDE.md: document the var-deref calling convention (public defns reached from the
  .ss runtime by string lookup aren't dead), the def-var! native pattern, and the
  overlay shadowing rule; point at MODULES.md.
- REFACTOR_PLAN.md: the prioritized, risk-tiered plan (working doc for this branch).
This commit is contained in:
Yogthos 2026-06-23 21:50:58 -04:00
parent 7d1f0b058c
commit d84c88f830
4 changed files with 242 additions and 0 deletions

View file

@ -109,6 +109,18 @@ Issue tracking and design notes live in beads (`bd prime`, `bd memories`).
forever. Use `jolt.host/ref-get`. forever. Use `jolt.host/ref-get`.
- **Map literals with `:jolt/type` as a key** parse as tagged reader forms — - **Map literals with `:jolt/type` as a key** parse as tagged reader forms —
don't tag overlay value maps in source. don't tag overlay value maps in source.
- **The compiler is reached from the runtime by `var-deref` string lookup.** The
`.ss` runtime calls into the cross-compiled compiler with
`(var-deref "jolt.analyzer" "analyze")` etc., and the compiler resolves its own
unqualified `jolt.host/…` refs the same way against `host-contract.ss`. So a
public `defn` with no in-Clojure callers may still be a live entry point — don't
treat it as dead. Only a private `defn-` with no callers is safe to remove.
- **A native `clojure.core` fn is a `(def-var! "clojure.core" "name" …)`** in a
`host/chez/*.ss`; the rest of core is the overlay (`jolt-core/clojure/core/*.clj`).
A few natives are re-asserted in `post-prelude.ss` so they win over the overlay.
See [docs/MODULES.md](docs/MODULES.md) for where a given fn lives, and
[docs/seed-overlay-registry.md](docs/seed-overlay-registry.md) for the shadowing
rule. Start at [docs/MODULES.md](docs/MODULES.md) to find a feature's files.
- **Fix latent bugs to match Clojure** rather than preserving them, with a - **Fix latent bugs to match Clojure** rather than preserving them, with a
regression case. Match the JVM (or provide a superset); the JVM-sourced corpus regression case. Match the JVM (or provide a superset); the JVM-sourced corpus
is the contract. is the contract.

117
REFACTOR_PLAN.md Normal file
View file

@ -0,0 +1,117 @@
# Architecture refactor plan
Working doc for the `spike/arch-refactor` branch. Goal: make the codebase easier
to understand and safely modify — optimized for both human and LLM maintainers.
Derived from a four-part review (compiler core, Chez runtime, build/tooling,
top-level organization) plus call-graph analysis of the compiler.
The codebase is in better shape than its 58-file runtime suggests: `rt.ss` is a
real load-order map, the `passes/` split is clean, filename prefixes encode tier
order, and the Makefile self-documents the gate. The work below targets the few
genuine liabilities.
## The re-mint constraint shapes risk
- **Seed sources** (`jolt-core/jolt/*.clj`, `jolt-core/clojure/core/*.clj`,
`host/chez/reader.ss`): a change needs `make remint` + the byte-fixpoint
`make selfhost`. Behaviour-preserving refactors are *verified* by selfhost, but
every touch costs a re-mint. Higher risk, batch them.
- **Runtime `.ss` + runtime-loaded `.clj`** (`build.ss`, `emit-image.ss`,
`main.clj`, the dispatcher shims, …): no re-mint; verified by `make test` +
`make shakesmoke`. Lower risk.
- **Docs / renames of non-seed files**: zero behaviour risk.
## Tier 0 — navigability (zero risk, highest LLM value, do first)
1. **`docs/MODULES.md` — the repo map.** One table: area → directory → key files →
re-mint? Plus a per-feature "touch points" list for cross-cutting features
(tree-shaking, direct-linking, numeric fl/fx, multimethods, the deps resolver).
Collapses the current 34-source lookup ("where does X live?") into one read.
2. **`docs/rfc/README.md`** — index the 7 RFCs (number, title, status), mirroring
`docs/spec/README.md`.
3. **Document the shipped-but-undocumented compiler features**: IR inlining and
fl*/fx* numeric lowering (the `passes/` pipeline). A short "compilation passes"
doc or an RFC, referenced from RFC 0007.
4. **CLAUDE.md — one "Invariants you must preserve" block**: the `var-deref`
calling convention (the compiler is reached from `.ss` by string lookup), the
`def-var!` native-binding pattern, and the seed dual-home shadowing rule (today
only in `seed-overlay-registry.md`).
5. **First-line purpose comment on every `.ss`** (audit the laggards) and note the
reader's surprising location (`host/chez/reader.ss`, Scheme, re-mint applies).
## Tier 1 — runtime structure (no re-mint; verify with `make test` + shakesmoke)
6. **Registry pattern for the core dispatchers — the biggest runtime win.**
`jolt-pr-str`, `jolt-pr-readable`, `jolt=2`, `jolt-get`, `jolt-class`,
`jolt-hash` are extended by `set!`-capture-and-rebind across ~8 files (e.g.
`jolt-pr-str` is wrapped 9× in 8 files). The registry pattern already exists in
tree (`register-str-render!`, `register-instance-check-arm!`) and is strictly
better: type-disjoint arms gathered in one walker, load-order-independent,
greppable. Add `register-pr-arm!` / `register-eq-arm!` / `register-get-arm!` /
`register-hash-arm!` / `register-class-arm!`; convert the ~35 `set!` sites to
one-line registrations. Eliminates "read 8 files to understand one dispatcher."
7. **Extract a `host/chez/dce.ss` module for tree-shaking.** The DCE logic is split
across `emit-image.ss` (the `dce-*` helpers + record producer) and `build.ss`
(`bld-shake-all` + the manifest splice). Pull every `dce-*` def + `bld-shake-all`
(→ `dce-shake`) into `dce.ss`; give the record a named accessor API
(`dce-rec-keep?`/`-fqn`/`-refs`/`-str`) instead of `(vector-ref r 0..3)`; split
`bld-shake-all`'s five jobs (root-seed, edge-build, BFS, bail-detect, partition)
into named steps. This is recently-added code and the loosest contract in the
build.
8. **Tag the runtime manifest.** `bld-emit-runtime` decides "splice shaken core" /
"drop compiler" by substring-matching `(load "…seed/prelude.ss")` strings.
Replace the manifest with tagged entries (`(prelude)`/`(image)`/`(compile-eval)`/
`(load path)`) and dispatch on the tag — removes a silent-failure coupling.
9. **`build-binary` options map.** It takes 8 positional args ending in two bare
booleans (`direct-link?` `tree-shake?`); a swap compiles and misbehaves silently.
Pass one options map from `main.clj`; new flags become additive.
10. **`dynamic-wind` the compiler-global set/reset** in `build-binary` so
`set-optimize!`/`set-direct-link!` always revert (today they leak on a build
error; harmless for the CLI, a trap for any in-process caller).
11. **Re-split `host-static-{,statics,objects}.ss`** on a real axis — static
methods/fields vs host object classes — instead of the current "grew too big"
chain (one file is 502 lines, the largest non-seed runtime file). Move the
System-property/env plumbing out to `io.ss` / a `host-system.ss`.
12. **Consolidate the seq/transducer/parity natives** and dissolve
`natives-parity.ss` (a literal leftovers file). One `natives-transduce.ss` for
`td-*`/`into-xform`/`transduce`/`sequence`/`cat`; relocate parity's hash /
macroexpand / reader-conditional pieces to their real homes.
13. **Smaller renames/moves:** `dynamic-vars.ss` (5 constants) vs `dyn-binding.ss`
(the binding stack) — rename the former; pull the `format` engine out of
`natives-misc.ss` into `natives-format.ss`.
## Tier 2 — compiler core (seed sources; re-mint + selfhost + corpus)
14. **Split the success-type checker out of `types.clj`** (716 lines = inference +
checker + driver). Move the checker (`check-*`, the error-domain predicates) to
`jolt.passes.types.check`. Highest reduction in edit blast-radius in the
compiler — inference and checker stop being able to break each other.
15. **Decompose `infer`'s `:invoke` arm** (≈120 lines, 8 hand-coded call patterns)
into named `infer-<pattern>` helpers (as `numeric.clj`'s `an-invoke` already
does), and add `ty`/`nd` accessors for the positional `[type node]` tuple
(a transposition is currently silent and type-correct).
16. **Single-source the const-keyword-lookup recognizer.** It's implemented three
times divergently (`inline.clj`, `types.clj` ×2 arms, `backend_scheme.clj`).
Lift one predicate into `jolt.passes.fold` (the shared-predicate home) for the
two analysis copies; the backend's value-emission copy can stay.
17. **Co-locate the numeric op tables.** `backend_scheme.clj` (`dbl-ops`/`lng-ops`,
the Scheme strings) and `numeric.clj` (`dbl-spec`/`lng-spec`, specializable
names) must agree or `emit-numeric` splices a `nil` op string. Cross-link or
share the name→op table.
18. **Collapse `local-escapes?`'s mechanical recursion** (≈12 of its arms are just
"does any child escape" = `reduce-ir-children`); keep only the binder/lookup
arms explicit. Removes ~50 lines and a new-op soundness hazard. Fold
`recur-kinds`/`recur-arg-lists` (numeric) into one `recur-tails`.
19. **Touch-when-nearby:** drop the dead `form-char?` refer in `analyzer.clj:21`;
standardize on `(get node :k)` in the pass files; promote a few inline-comment
soundness arguments to function docstrings (`an-invoke` `:wild`, `dbl/lng-spec`).
## Sequencing
Tier 0 lands immediately (docs only). Tier 1 is the bulk of the maintainability win
and carries no re-mint — do 6 (registry) and 710 (DCE module + build hygiene)
first; they target the code most likely to be edited and most recently grown.
Tier 2 batches into one or two re-mints once Tier 1 is stable, each verified by
`make selfhost` + `make corpus`.
Every code change in Tiers 12 is behaviour-preserving and gated; no feature work.

92
docs/MODULES.md Normal file
View file

@ -0,0 +1,92 @@
# Module map
Where things live and what to read before changing them. Start here to answer
"where does feature X live?" and "what else do I need to touch?"
## Areas
| Area | Directory | Responsibility | Re-mint? |
| --- | --- | --- | --- |
| Chez runtime | `host/chez/*.ss` | The substrate: value model, persistent collections, seqs, vars/namespaces, host interop, native `clojure.core` shims, regex, FFI, IO, the **reader**. Composed by `rt.ss`. | only `reader.ss` |
| Compiler | `jolt-core/jolt/*.clj` | analyzer → IR → backend, the optimization passes, the CLI, the deps resolver, nREPL. Baked into the seed. | **yes** |
| `clojure.core` overlay | `jolt-core/clojure/core/NN-*.clj` | Portable `clojure.core` in dependency-ordered tiers (`00-syntax``50-io`); the `NN` prefix *is* the load order. | **yes** |
| Stdlib | `stdlib/clojure/*.clj` | Lazily-loaded portable namespaces (string/set/walk/edn/pprint/zip/test/data). | no |
| Build & tooling | `host/chez/build.ss`, `emit-image.ss`, `compile-eval.ss`, `loader.ss`, `cli.ss`, `bootstrap.ss` | AOT binary build, cross-compile, runtime eval/load, CLI spine, seed mint. | no (except via `reader.ss`) |
| Tests & gate | `test/chez/`, `test/conformance/`, `host/chez/run-*.ss`, `Makefile` | Corpus (JVM oracle), unit, per-feature tests. Every `make` target has a comment. | no |
**The reader is in `host/chez/reader.ss`** (Scheme, a seed source) — *not* in
`jolt-core/jolt/` with the rest of the compiler. Re-mint applies to it.
`rt.ss` is the runtime's load-order manifest: it `(load …)`s every shim in
dependency order with a per-file comment. Read it to see how the runtime is
composed and where a given `.ss` fits.
## `host/chez/*.ss` by family
- **Value model**: `values.ss` (nil/numbers/keywords/symbols), `collections.ss`
(persistent vec + HAMT map/set), `seq.ss` + `lazy-bridge.ss` (seqs, lazy-seqs),
`transients.ss`, `records.ss` + `records-interop.ss`.
- **Native `clojure.core` shims**: `natives-*.ss` (array/coll/meta/misc/num/parity/
queue/seq/str/xform), plus `predicates.ss`, `converters.ss`, `printing.ss`.
- **Vars / namespaces / dynamics**: `vars.ss`, `ns.ss`, `dyn-binding.ss` (the
thread-local binding stack), `dynamic-vars.ss` (a few `*…*` constant defaults),
`atoms.ss`, `multimethods.ss`.
- **Host interop**: `host-class.ss` (class tokens + method dispatch),
`host-static*.ss` (static methods/fields + host object classes), `host-table.ss`,
`host-contract.ss` (the `jolt.host` seam the compiler resolves against),
`dot-forms.ss`, `records-interop.ss`.
- **Scalars / misc**: `regex.ss` (vendored irregex), `math.ss`, `inst-time.ss`,
`bigdec.ss`, `syntax-quote.ss`.
- **IO / system / concurrency / FFI**: `io.ss`, `png.ss`, `concurrency.ss`,
`async.ss`, `ffi.ss`.
- **Compiler entry on Chez**: `reader.ss`, `compile-eval.ss`, `emit-image.ss`,
`loader.ss`, `cli.ss`, `build.ss`, `bootstrap.ss`.
## Where is a `clojure.core` fn implemented?
Two homes, with a defined precedence:
1. **Native shim** — a `(def-var! "clojure.core" "name" …)` in a `host/chez/*.ss`
(hot/representation-coupled fns: `first`, `get`, `=`, the predicates).
2. **Overlay** — a `defn` in a `jolt-core/clojure/core/NN-*.clj` tier (most of
`clojure.core`, in portable Clojure).
3. **`post-prelude.ss`** re-asserts a handful of natives *after* the overlay loads,
so the native version wins (the overlay's value-reading versions are wrong for
Chez-native chars/atoms/etc.). Each entry there says why.
`grep 'def-var! "clojure.core" "frequencies"' host/chez` and
`grep -rn 'defn frequencies' jolt-core/clojure/core` to find a given fn. See
[seed-overlay-registry.md](seed-overlay-registry.md) for the shadowing mechanism.
## Cross-cutting features — touch points
A feature's *core* lives in one file; these are the other files you must keep in
sync when changing it.
- **Tree-shaking / DCE** (`--tree-shake`): `emit-image.ss` (the `dce-*` helpers +
record producers) and `build.ss` (`bld-shake-all` reachability + the manifest
splice in `bld-emit-runtime`); the flag in `main.clj`; validated by
`host/chez/tree-shake-smoke.sh` (`make shakesmoke`) and `build-smoke.sh`. See
[tools-deps.md](tools-deps.md#tree-shaking).
- **Direct-linking** (`--direct-link`): `backend_scheme.clj` (`direct-link?`,
`emit-top-form`, the `jv$<fqn>` bindings); `build.ss` turns it on; `main.clj` the
flag; `test/chez/directlink-test.ss`.
- **Numeric fl*/fx\*** (`^double`/`^long` hints): `jolt-core/jolt/passes/numeric.clj`
(the hint-directed pass + loop-counter + `:coerce`); `backend_scheme.clj`
(`dbl-ops`/`lng-ops` op strings, `emit-numeric`, entry/return coercion);
`analyzer.clj` (`nhint-of`, `:nhints`, `with-ret-nhint`); `host-contract.ss`
(`:num-ret` on resolve); `rt.ss` (`jolt->fx`); `test/chez/numeric-test.ss`.
- **IR inlining** (under `--opt`): `passes/inline.clj` (splice) + `passes.clj`
(stash) + `host-contract.ss` (`inline-ir`/`stash-inline!`); `test/chez/inline-test.ss`.
- **Multimethods**: `host/chez/multimethods.ss` (dispatch) + the overlay
`defmulti`/`defmethod` macros + `host-contract.ss` late-bind.
- **Deps resolution**: `jolt-core/jolt/deps.clj` (the only file) + `main.clj`
(applies the roots) + `loader.ss` (the `require` path).
## Conventions you must preserve
See **CLAUDE.md → "Conventions & Patterns"** for the load-bearing rules: the
re-mint trigger, the tier macro-ordering rule, the `get`-on-your-own-wrapper trap,
`:jolt/type`-as-a-key parsing, the `var-deref` calling convention (the compiler is
reached from the `.ss` runtime by string lookup, so a public `defn` with no
in-Clojure callers can still be live), and the writing style.

21
docs/rfc/README.md Normal file
View file

@ -0,0 +1,21 @@
# RFCs
Design notes for non-obvious language and compiler decisions. An RFC records *why*
a thing is built the way it is; the code is the source of truth for *how*.
| # | Title | Status | Governs |
| --- | --- | --- | --- |
| [0001](0001-language-specification.md) | A Specification for the Clojure Language | Draft | The conformance target — what "is Clojure" means for jolt. |
| [0002](0002-reader-conditional-features.md) | Reader-Conditional Feature Set | Accepted | `#?(...)` feature keys (`:jolt`, `:clj`, `:default`). |
| [0003](0003-transients.md) | Transients | Accepted | `transient`/`persistent!` semantics + the Chez mutable backing. |
| [0004](0004-type-hints.md) | Type hints + keyword-lookup specialization | Accepted | `^Type`/`^:struct` hints → the bare-`get` fast path. |
| [0005](0005-structural-type-inference.md) | Structural collection-type inference | Implemented | The `:struct`/`:vec`/`:set` lattice in `passes/types`. |
| [0006](0006-success-type-checking.md) | Success typing (provably-wrong-code detection) | Implemented | The error-domain checker in `passes/types`. |
| [0007](0007-compilation-modes-and-binary-output.md) | Compilation modes + binary output | Implemented (doc lags) | `release`/`--opt`/`--dev`, `--direct-link`, `--tree-shake`. |
RFC 0007's own status line still says "Draft, no code yet" — that is stale:
direct-linking and tree-shaking shipped (see [tools-deps.md](../tools-deps.md) and
`backend_scheme.clj` / `build.ss`). Two compiler features that grew alongside it —
**IR inlining** (`passes/inline.clj`, under `--opt`) and **numeric `fl*`/`fx*`
lowering** from `^double`/`^long` hints (`passes/numeric.clj`) — are not yet written
up as RFCs; their touch points are in [../MODULES.md](../MODULES.md).