docs: module map, RFC index, refactor plan (arch-refactor tier 0)
Navigability groundwork from the architecture review — zero behaviour change. - docs/MODULES.md: the repo map. Area -> directory -> key files -> re-mint?, plus per-feature touch points (tree-shaking, direct-linking, numeric fl/fx, inlining, multimethods, deps) and where a given clojure.core fn lives. Answers "where does X live / what's related to Y" in one read. - docs/rfc/README.md: index the 7 RFCs; flags RFC 0007's stale "no code yet" status (direct-linking + tree-shaking shipped) and the undocumented inlining/numeric work. - CLAUDE.md: document the var-deref calling convention (public defns reached from the .ss runtime by string lookup aren't dead), the def-var! native pattern, and the overlay shadowing rule; point at MODULES.md. - REFACTOR_PLAN.md: the prioritized, risk-tiered plan (working doc for this branch).
This commit is contained in:
parent
7d1f0b058c
commit
d84c88f830
4 changed files with 242 additions and 0 deletions
12
CLAUDE.md
12
CLAUDE.md
|
|
@ -109,6 +109,18 @@ Issue tracking and design notes live in beads (`bd prime`, `bd memories`).
|
||||||
forever. Use `jolt.host/ref-get`.
|
forever. Use `jolt.host/ref-get`.
|
||||||
- **Map literals with `:jolt/type` as a key** parse as tagged reader forms —
|
- **Map literals with `:jolt/type` as a key** parse as tagged reader forms —
|
||||||
don't tag overlay value maps in source.
|
don't tag overlay value maps in source.
|
||||||
|
- **The compiler is reached from the runtime by `var-deref` string lookup.** The
|
||||||
|
`.ss` runtime calls into the cross-compiled compiler with
|
||||||
|
`(var-deref "jolt.analyzer" "analyze")` etc., and the compiler resolves its own
|
||||||
|
unqualified `jolt.host/…` refs the same way against `host-contract.ss`. So a
|
||||||
|
public `defn` with no in-Clojure callers may still be a live entry point — don't
|
||||||
|
treat it as dead. Only a private `defn-` with no callers is safe to remove.
|
||||||
|
- **A native `clojure.core` fn is a `(def-var! "clojure.core" "name" …)`** in a
|
||||||
|
`host/chez/*.ss`; the rest of core is the overlay (`jolt-core/clojure/core/*.clj`).
|
||||||
|
A few natives are re-asserted in `post-prelude.ss` so they win over the overlay.
|
||||||
|
See [docs/MODULES.md](docs/MODULES.md) for where a given fn lives, and
|
||||||
|
[docs/seed-overlay-registry.md](docs/seed-overlay-registry.md) for the shadowing
|
||||||
|
rule. Start at [docs/MODULES.md](docs/MODULES.md) to find a feature's files.
|
||||||
- **Fix latent bugs to match Clojure** rather than preserving them, with a
|
- **Fix latent bugs to match Clojure** rather than preserving them, with a
|
||||||
regression case. Match the JVM (or provide a superset); the JVM-sourced corpus
|
regression case. Match the JVM (or provide a superset); the JVM-sourced corpus
|
||||||
is the contract.
|
is the contract.
|
||||||
|
|
|
||||||
117
REFACTOR_PLAN.md
Normal file
117
REFACTOR_PLAN.md
Normal file
|
|
@ -0,0 +1,117 @@
|
||||||
|
# Architecture refactor plan
|
||||||
|
|
||||||
|
Working doc for the `spike/arch-refactor` branch. Goal: make the codebase easier
|
||||||
|
to understand and safely modify — optimized for both human and LLM maintainers.
|
||||||
|
Derived from a four-part review (compiler core, Chez runtime, build/tooling,
|
||||||
|
top-level organization) plus call-graph analysis of the compiler.
|
||||||
|
|
||||||
|
The codebase is in better shape than its 58-file runtime suggests: `rt.ss` is a
|
||||||
|
real load-order map, the `passes/` split is clean, filename prefixes encode tier
|
||||||
|
order, and the Makefile self-documents the gate. The work below targets the few
|
||||||
|
genuine liabilities.
|
||||||
|
|
||||||
|
## The re-mint constraint shapes risk
|
||||||
|
|
||||||
|
- **Seed sources** (`jolt-core/jolt/*.clj`, `jolt-core/clojure/core/*.clj`,
|
||||||
|
`host/chez/reader.ss`): a change needs `make remint` + the byte-fixpoint
|
||||||
|
`make selfhost`. Behaviour-preserving refactors are *verified* by selfhost, but
|
||||||
|
every touch costs a re-mint. Higher risk, batch them.
|
||||||
|
- **Runtime `.ss` + runtime-loaded `.clj`** (`build.ss`, `emit-image.ss`,
|
||||||
|
`main.clj`, the dispatcher shims, …): no re-mint; verified by `make test` +
|
||||||
|
`make shakesmoke`. Lower risk.
|
||||||
|
- **Docs / renames of non-seed files**: zero behaviour risk.
|
||||||
|
|
||||||
|
## Tier 0 — navigability (zero risk, highest LLM value, do first)
|
||||||
|
|
||||||
|
1. **`docs/MODULES.md` — the repo map.** One table: area → directory → key files →
|
||||||
|
re-mint? Plus a per-feature "touch points" list for cross-cutting features
|
||||||
|
(tree-shaking, direct-linking, numeric fl/fx, multimethods, the deps resolver).
|
||||||
|
Collapses the current 3–4-source lookup ("where does X live?") into one read.
|
||||||
|
2. **`docs/rfc/README.md`** — index the 7 RFCs (number, title, status), mirroring
|
||||||
|
`docs/spec/README.md`.
|
||||||
|
3. **Document the shipped-but-undocumented compiler features**: IR inlining and
|
||||||
|
fl*/fx* numeric lowering (the `passes/` pipeline). A short "compilation passes"
|
||||||
|
doc or an RFC, referenced from RFC 0007.
|
||||||
|
4. **CLAUDE.md — one "Invariants you must preserve" block**: the `var-deref`
|
||||||
|
calling convention (the compiler is reached from `.ss` by string lookup), the
|
||||||
|
`def-var!` native-binding pattern, and the seed dual-home shadowing rule (today
|
||||||
|
only in `seed-overlay-registry.md`).
|
||||||
|
5. **First-line purpose comment on every `.ss`** (audit the laggards) and note the
|
||||||
|
reader's surprising location (`host/chez/reader.ss`, Scheme, re-mint applies).
|
||||||
|
|
||||||
|
## Tier 1 — runtime structure (no re-mint; verify with `make test` + shakesmoke)
|
||||||
|
|
||||||
|
6. **Registry pattern for the core dispatchers — the biggest runtime win.**
|
||||||
|
`jolt-pr-str`, `jolt-pr-readable`, `jolt=2`, `jolt-get`, `jolt-class`,
|
||||||
|
`jolt-hash` are extended by `set!`-capture-and-rebind across ~8 files (e.g.
|
||||||
|
`jolt-pr-str` is wrapped 9× in 8 files). The registry pattern already exists in
|
||||||
|
tree (`register-str-render!`, `register-instance-check-arm!`) and is strictly
|
||||||
|
better: type-disjoint arms gathered in one walker, load-order-independent,
|
||||||
|
greppable. Add `register-pr-arm!` / `register-eq-arm!` / `register-get-arm!` /
|
||||||
|
`register-hash-arm!` / `register-class-arm!`; convert the ~35 `set!` sites to
|
||||||
|
one-line registrations. Eliminates "read 8 files to understand one dispatcher."
|
||||||
|
7. **Extract a `host/chez/dce.ss` module for tree-shaking.** The DCE logic is split
|
||||||
|
across `emit-image.ss` (the `dce-*` helpers + record producer) and `build.ss`
|
||||||
|
(`bld-shake-all` + the manifest splice). Pull every `dce-*` def + `bld-shake-all`
|
||||||
|
(→ `dce-shake`) into `dce.ss`; give the record a named accessor API
|
||||||
|
(`dce-rec-keep?`/`-fqn`/`-refs`/`-str`) instead of `(vector-ref r 0..3)`; split
|
||||||
|
`bld-shake-all`'s five jobs (root-seed, edge-build, BFS, bail-detect, partition)
|
||||||
|
into named steps. This is recently-added code and the loosest contract in the
|
||||||
|
build.
|
||||||
|
8. **Tag the runtime manifest.** `bld-emit-runtime` decides "splice shaken core" /
|
||||||
|
"drop compiler" by substring-matching `(load "…seed/prelude.ss")` strings.
|
||||||
|
Replace the manifest with tagged entries (`(prelude)`/`(image)`/`(compile-eval)`/
|
||||||
|
`(load path)`) and dispatch on the tag — removes a silent-failure coupling.
|
||||||
|
9. **`build-binary` options map.** It takes 8 positional args ending in two bare
|
||||||
|
booleans (`direct-link?` `tree-shake?`); a swap compiles and misbehaves silently.
|
||||||
|
Pass one options map from `main.clj`; new flags become additive.
|
||||||
|
10. **`dynamic-wind` the compiler-global set/reset** in `build-binary` so
|
||||||
|
`set-optimize!`/`set-direct-link!` always revert (today they leak on a build
|
||||||
|
error; harmless for the CLI, a trap for any in-process caller).
|
||||||
|
11. **Re-split `host-static-{,statics,objects}.ss`** on a real axis — static
|
||||||
|
methods/fields vs host object classes — instead of the current "grew too big"
|
||||||
|
chain (one file is 502 lines, the largest non-seed runtime file). Move the
|
||||||
|
System-property/env plumbing out to `io.ss` / a `host-system.ss`.
|
||||||
|
12. **Consolidate the seq/transducer/parity natives** and dissolve
|
||||||
|
`natives-parity.ss` (a literal leftovers file). One `natives-transduce.ss` for
|
||||||
|
`td-*`/`into-xform`/`transduce`/`sequence`/`cat`; relocate parity's hash /
|
||||||
|
macroexpand / reader-conditional pieces to their real homes.
|
||||||
|
13. **Smaller renames/moves:** `dynamic-vars.ss` (5 constants) vs `dyn-binding.ss`
|
||||||
|
(the binding stack) — rename the former; pull the `format` engine out of
|
||||||
|
`natives-misc.ss` into `natives-format.ss`.
|
||||||
|
|
||||||
|
## Tier 2 — compiler core (seed sources; re-mint + selfhost + corpus)
|
||||||
|
|
||||||
|
14. **Split the success-type checker out of `types.clj`** (716 lines = inference +
|
||||||
|
checker + driver). Move the checker (`check-*`, the error-domain predicates) to
|
||||||
|
`jolt.passes.types.check`. Highest reduction in edit blast-radius in the
|
||||||
|
compiler — inference and checker stop being able to break each other.
|
||||||
|
15. **Decompose `infer`'s `:invoke` arm** (≈120 lines, 8 hand-coded call patterns)
|
||||||
|
into named `infer-<pattern>` helpers (as `numeric.clj`'s `an-invoke` already
|
||||||
|
does), and add `ty`/`nd` accessors for the positional `[type node]` tuple
|
||||||
|
(a transposition is currently silent and type-correct).
|
||||||
|
16. **Single-source the const-keyword-lookup recognizer.** It's implemented three
|
||||||
|
times divergently (`inline.clj`, `types.clj` ×2 arms, `backend_scheme.clj`).
|
||||||
|
Lift one predicate into `jolt.passes.fold` (the shared-predicate home) for the
|
||||||
|
two analysis copies; the backend's value-emission copy can stay.
|
||||||
|
17. **Co-locate the numeric op tables.** `backend_scheme.clj` (`dbl-ops`/`lng-ops`,
|
||||||
|
the Scheme strings) and `numeric.clj` (`dbl-spec`/`lng-spec`, specializable
|
||||||
|
names) must agree or `emit-numeric` splices a `nil` op string. Cross-link or
|
||||||
|
share the name→op table.
|
||||||
|
18. **Collapse `local-escapes?`'s mechanical recursion** (≈12 of its arms are just
|
||||||
|
"does any child escape" = `reduce-ir-children`); keep only the binder/lookup
|
||||||
|
arms explicit. Removes ~50 lines and a new-op soundness hazard. Fold
|
||||||
|
`recur-kinds`/`recur-arg-lists` (numeric) into one `recur-tails`.
|
||||||
|
19. **Touch-when-nearby:** drop the dead `form-char?` refer in `analyzer.clj:21`;
|
||||||
|
standardize on `(get node :k)` in the pass files; promote a few inline-comment
|
||||||
|
soundness arguments to function docstrings (`an-invoke` `:wild`, `dbl/lng-spec`).
|
||||||
|
|
||||||
|
## Sequencing
|
||||||
|
|
||||||
|
Tier 0 lands immediately (docs only). Tier 1 is the bulk of the maintainability win
|
||||||
|
and carries no re-mint — do 6 (registry) and 7–10 (DCE module + build hygiene)
|
||||||
|
first; they target the code most likely to be edited and most recently grown.
|
||||||
|
Tier 2 batches into one or two re-mints once Tier 1 is stable, each verified by
|
||||||
|
`make selfhost` + `make corpus`.
|
||||||
|
|
||||||
|
Every code change in Tiers 1–2 is behaviour-preserving and gated; no feature work.
|
||||||
92
docs/MODULES.md
Normal file
92
docs/MODULES.md
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
# Module map
|
||||||
|
|
||||||
|
Where things live and what to read before changing them. Start here to answer
|
||||||
|
"where does feature X live?" and "what else do I need to touch?"
|
||||||
|
|
||||||
|
## Areas
|
||||||
|
|
||||||
|
| Area | Directory | Responsibility | Re-mint? |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Chez runtime | `host/chez/*.ss` | The substrate: value model, persistent collections, seqs, vars/namespaces, host interop, native `clojure.core` shims, regex, FFI, IO, the **reader**. Composed by `rt.ss`. | only `reader.ss` |
|
||||||
|
| Compiler | `jolt-core/jolt/*.clj` | analyzer → IR → backend, the optimization passes, the CLI, the deps resolver, nREPL. Baked into the seed. | **yes** |
|
||||||
|
| `clojure.core` overlay | `jolt-core/clojure/core/NN-*.clj` | Portable `clojure.core` in dependency-ordered tiers (`00-syntax` … `50-io`); the `NN` prefix *is* the load order. | **yes** |
|
||||||
|
| Stdlib | `stdlib/clojure/*.clj` | Lazily-loaded portable namespaces (string/set/walk/edn/pprint/zip/test/data). | no |
|
||||||
|
| Build & tooling | `host/chez/build.ss`, `emit-image.ss`, `compile-eval.ss`, `loader.ss`, `cli.ss`, `bootstrap.ss` | AOT binary build, cross-compile, runtime eval/load, CLI spine, seed mint. | no (except via `reader.ss`) |
|
||||||
|
| Tests & gate | `test/chez/`, `test/conformance/`, `host/chez/run-*.ss`, `Makefile` | Corpus (JVM oracle), unit, per-feature tests. Every `make` target has a comment. | no |
|
||||||
|
|
||||||
|
**The reader is in `host/chez/reader.ss`** (Scheme, a seed source) — *not* in
|
||||||
|
`jolt-core/jolt/` with the rest of the compiler. Re-mint applies to it.
|
||||||
|
|
||||||
|
`rt.ss` is the runtime's load-order manifest: it `(load …)`s every shim in
|
||||||
|
dependency order with a per-file comment. Read it to see how the runtime is
|
||||||
|
composed and where a given `.ss` fits.
|
||||||
|
|
||||||
|
## `host/chez/*.ss` by family
|
||||||
|
|
||||||
|
- **Value model**: `values.ss` (nil/numbers/keywords/symbols), `collections.ss`
|
||||||
|
(persistent vec + HAMT map/set), `seq.ss` + `lazy-bridge.ss` (seqs, lazy-seqs),
|
||||||
|
`transients.ss`, `records.ss` + `records-interop.ss`.
|
||||||
|
- **Native `clojure.core` shims**: `natives-*.ss` (array/coll/meta/misc/num/parity/
|
||||||
|
queue/seq/str/xform), plus `predicates.ss`, `converters.ss`, `printing.ss`.
|
||||||
|
- **Vars / namespaces / dynamics**: `vars.ss`, `ns.ss`, `dyn-binding.ss` (the
|
||||||
|
thread-local binding stack), `dynamic-vars.ss` (a few `*…*` constant defaults),
|
||||||
|
`atoms.ss`, `multimethods.ss`.
|
||||||
|
- **Host interop**: `host-class.ss` (class tokens + method dispatch),
|
||||||
|
`host-static*.ss` (static methods/fields + host object classes), `host-table.ss`,
|
||||||
|
`host-contract.ss` (the `jolt.host` seam the compiler resolves against),
|
||||||
|
`dot-forms.ss`, `records-interop.ss`.
|
||||||
|
- **Scalars / misc**: `regex.ss` (vendored irregex), `math.ss`, `inst-time.ss`,
|
||||||
|
`bigdec.ss`, `syntax-quote.ss`.
|
||||||
|
- **IO / system / concurrency / FFI**: `io.ss`, `png.ss`, `concurrency.ss`,
|
||||||
|
`async.ss`, `ffi.ss`.
|
||||||
|
- **Compiler entry on Chez**: `reader.ss`, `compile-eval.ss`, `emit-image.ss`,
|
||||||
|
`loader.ss`, `cli.ss`, `build.ss`, `bootstrap.ss`.
|
||||||
|
|
||||||
|
## Where is a `clojure.core` fn implemented?
|
||||||
|
|
||||||
|
Two homes, with a defined precedence:
|
||||||
|
|
||||||
|
1. **Native shim** — a `(def-var! "clojure.core" "name" …)` in a `host/chez/*.ss`
|
||||||
|
(hot/representation-coupled fns: `first`, `get`, `=`, the predicates).
|
||||||
|
2. **Overlay** — a `defn` in a `jolt-core/clojure/core/NN-*.clj` tier (most of
|
||||||
|
`clojure.core`, in portable Clojure).
|
||||||
|
3. **`post-prelude.ss`** re-asserts a handful of natives *after* the overlay loads,
|
||||||
|
so the native version wins (the overlay's value-reading versions are wrong for
|
||||||
|
Chez-native chars/atoms/etc.). Each entry there says why.
|
||||||
|
|
||||||
|
`grep 'def-var! "clojure.core" "frequencies"' host/chez` and
|
||||||
|
`grep -rn 'defn frequencies' jolt-core/clojure/core` to find a given fn. See
|
||||||
|
[seed-overlay-registry.md](seed-overlay-registry.md) for the shadowing mechanism.
|
||||||
|
|
||||||
|
## Cross-cutting features — touch points
|
||||||
|
|
||||||
|
A feature's *core* lives in one file; these are the other files you must keep in
|
||||||
|
sync when changing it.
|
||||||
|
|
||||||
|
- **Tree-shaking / DCE** (`--tree-shake`): `emit-image.ss` (the `dce-*` helpers +
|
||||||
|
record producers) and `build.ss` (`bld-shake-all` reachability + the manifest
|
||||||
|
splice in `bld-emit-runtime`); the flag in `main.clj`; validated by
|
||||||
|
`host/chez/tree-shake-smoke.sh` (`make shakesmoke`) and `build-smoke.sh`. See
|
||||||
|
[tools-deps.md](tools-deps.md#tree-shaking).
|
||||||
|
- **Direct-linking** (`--direct-link`): `backend_scheme.clj` (`direct-link?`,
|
||||||
|
`emit-top-form`, the `jv$<fqn>` bindings); `build.ss` turns it on; `main.clj` the
|
||||||
|
flag; `test/chez/directlink-test.ss`.
|
||||||
|
- **Numeric fl*/fx\*** (`^double`/`^long` hints): `jolt-core/jolt/passes/numeric.clj`
|
||||||
|
(the hint-directed pass + loop-counter + `:coerce`); `backend_scheme.clj`
|
||||||
|
(`dbl-ops`/`lng-ops` op strings, `emit-numeric`, entry/return coercion);
|
||||||
|
`analyzer.clj` (`nhint-of`, `:nhints`, `with-ret-nhint`); `host-contract.ss`
|
||||||
|
(`:num-ret` on resolve); `rt.ss` (`jolt->fx`); `test/chez/numeric-test.ss`.
|
||||||
|
- **IR inlining** (under `--opt`): `passes/inline.clj` (splice) + `passes.clj`
|
||||||
|
(stash) + `host-contract.ss` (`inline-ir`/`stash-inline!`); `test/chez/inline-test.ss`.
|
||||||
|
- **Multimethods**: `host/chez/multimethods.ss` (dispatch) + the overlay
|
||||||
|
`defmulti`/`defmethod` macros + `host-contract.ss` late-bind.
|
||||||
|
- **Deps resolution**: `jolt-core/jolt/deps.clj` (the only file) + `main.clj`
|
||||||
|
(applies the roots) + `loader.ss` (the `require` path).
|
||||||
|
|
||||||
|
## Conventions you must preserve
|
||||||
|
|
||||||
|
See **CLAUDE.md → "Conventions & Patterns"** for the load-bearing rules: the
|
||||||
|
re-mint trigger, the tier macro-ordering rule, the `get`-on-your-own-wrapper trap,
|
||||||
|
`:jolt/type`-as-a-key parsing, the `var-deref` calling convention (the compiler is
|
||||||
|
reached from the `.ss` runtime by string lookup, so a public `defn` with no
|
||||||
|
in-Clojure callers can still be live), and the writing style.
|
||||||
21
docs/rfc/README.md
Normal file
21
docs/rfc/README.md
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
# RFCs
|
||||||
|
|
||||||
|
Design notes for non-obvious language and compiler decisions. An RFC records *why*
|
||||||
|
a thing is built the way it is; the code is the source of truth for *how*.
|
||||||
|
|
||||||
|
| # | Title | Status | Governs |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| [0001](0001-language-specification.md) | A Specification for the Clojure Language | Draft | The conformance target — what "is Clojure" means for jolt. |
|
||||||
|
| [0002](0002-reader-conditional-features.md) | Reader-Conditional Feature Set | Accepted | `#?(...)` feature keys (`:jolt`, `:clj`, `:default`). |
|
||||||
|
| [0003](0003-transients.md) | Transients | Accepted | `transient`/`persistent!` semantics + the Chez mutable backing. |
|
||||||
|
| [0004](0004-type-hints.md) | Type hints + keyword-lookup specialization | Accepted | `^Type`/`^:struct` hints → the bare-`get` fast path. |
|
||||||
|
| [0005](0005-structural-type-inference.md) | Structural collection-type inference | Implemented | The `:struct`/`:vec`/`:set` lattice in `passes/types`. |
|
||||||
|
| [0006](0006-success-type-checking.md) | Success typing (provably-wrong-code detection) | Implemented | The error-domain checker in `passes/types`. |
|
||||||
|
| [0007](0007-compilation-modes-and-binary-output.md) | Compilation modes + binary output | Implemented (doc lags) | `release`/`--opt`/`--dev`, `--direct-link`, `--tree-shake`. |
|
||||||
|
|
||||||
|
RFC 0007's own status line still says "Draft, no code yet" — that is stale:
|
||||||
|
direct-linking and tree-shaking shipped (see [tools-deps.md](../tools-deps.md) and
|
||||||
|
`backend_scheme.clj` / `build.ss`). Two compiler features that grew alongside it —
|
||||||
|
**IR inlining** (`passes/inline.clj`, under `--opt`) and **numeric `fl*`/`fx*`
|
||||||
|
lowering** from `^double`/`^long` hints (`passes/numeric.clj`) — are not yet written
|
||||||
|
up as RFCs; their touch points are in [../MODULES.md](../MODULES.md).
|
||||||
Loading…
Add table
Add a link
Reference in a new issue