jolt/docs/architecture-refactor-plan.md

# Architecture Refactor Plan

Goal: make the jolt codebase easier to understand and safe to change — for human
maintainers and, specifically, for LLM agents. An agent should be able to find
where a feature lives, see its related code, and make a change without scanning
3000-line files or keeping invisible global state in mind.

This plan synthesizes a six-part architectural review (one reviewer per
subsystem). It is organized as **independent, gate-validated phases**, ordered by
value-to-risk. Each phase is a PR-sized unit. Nothing here changes behavior; it is
pure reorganization plus a small set of dead-code/bug deletions.

## Non-negotiable constraints (apply to every phase)

- **Every phase passes the full gate**: `jpm test` green (conformance ×3 modes,
  `clojure-test-suite` ≥ baseline, bench back-to-back vs main, real exit code).
  `rm -rf build && jpm clean` before trusting the binary.
- **Load order is load-bearing.** The Janet seed (`src/jolt/*.janet`) and the
  Clojure overlay tiers (`jolt-core/clojure/core/NN-*.clj`) load in a fixed order;
  a module may only use what loads before it. Splitting a file must preserve the
  net load order (a new file is imported where the old code ran).
- **Seed-tier discipline** (the jolt-tzo rules in CLAUDE.md): nothing the
  analyzer/IR use may move below the kernel tier; a tier may only use macros from
  earlier tiers; expander-called fns stay in 00-syntax.
- **One concern per PR.** Do not combine a file split with a behavior fix.
  Dead-code/bug deletions (Phase 0) land first and separately so later diffs are
  pure moves.

## Guiding principles (the target state)

1. **One obvious home per feature.** Adding a `.someMethod` interop shim, a reader
   macro, or a clojure.core fn should have exactly one file an agent edits.
2. **Files map to concerns, not to history.** No 3000-line grab-bags; no module
   whose name lies about its contents (`javatime.janet` is 80% not java.time;
   `phm.janet` contains LazySeq; `types.janet` holds seven concerns).
3. **Make implicit contracts explicit and checked.** The seed↔overlay split, the
   ctx-shaping env-knob list, and the IR op set are tribal knowledge today; turn
   each into a single source of truth with a drift check.
4. **No copy-paste dispatch.** Where the same op-set / member-dispatch / cache
   dance is hand-written N times, extract one combinator.

---

## Phase 0 — Dead code & concrete bugs (low risk, do first)

Pure deletions and small fixes, each with a regression row. These remove
*actively misleading* code (comments that contradict behavior) before any moves.

| Item | Location | Note |
|---|---|---|
| `find` defined twice; the dead copy returns a plain vector and is *wrong* | `jolt-core/clojure/core/20-coll.clj:347-349` | live def at :787 returns a real map-entry; delete the dead one |
| `core-satisfies?` always returns `false` (latent bug + misleading comment) | `src/jolt/core.janet:2412` | either implement over the protocol registry or document why inert; fix the `eval-list` comment that claims it's an overlay fn |
| `File.toURL` stores `:url` but every `:jolt/url` method reads `:spec` | `src/jolt/javatime.janet:636` | broken shim; use `:spec`, add a spec row |
| `core-type->str` — zero references | `src/jolt/core.janet:2416` | delete defn + binding |
| `core-resolve` — unreachable (overridden by `install-stateful-fns!`) | `src/jolt/core.janet:2365` | delete defn + binding; fix stale comment |
| `mark-hint` — unreachable | `jolt-core/jolt/passes.clj:836` | delete |
| `pad2` defined twice | `src/jolt/javatime.janet:41,522` | keep one |
| `phs-to-struct`, `shape-vals`, `ns-imports-fn` — zero references | `phm.janet:302`, `types.janet:655`, `types.janet:527` | delete unless reserved API |
| `pl-rest` no-op `(if (plist? r) r r)` | `src/jolt/plist.janet:62` | collapse / fix; regression check |
| `read-quote` unused `pos` param | `src/jolt/reader.janet:608` | drop |
| Stale/contradictory comments (`extend`, orphan `;; trampoline:` / `;; rand-int:` headers, migration breadcrumbs) | `30-macros.clj:402-404`, `20-coll.clj:558-560,780,790` | sweep |
| **`:map-shapes?` missing from the deps-image cache key** (possible stale-image correctness gap) | `src/jolt/main.janet:440-448` | add to key; confirm with a test |

---

## Phase 1 — Extract the host-interop subsystem (highest value)

**Problem (corroborated by 3 reviewers).** The JVM-emulation shim layer is the
single worst sprawl, and it is exactly where the recent hiccup/markdown/malli work
landed ad hoc. A single class is split across up to three files:

- registry *machinery* (`class-statics`/`tagged-methods`/`class-ctors` +
  `register-*!`) lives in `evaluator.janet:624-651`;
- four static tables (`Math`/`Thread`/`System`/`Long`/`Number`) are hardcoded in
  `evaluator.janet:537-619` and dispatched *outside* the registry in `resolve-sym`
  (780-794); instance-method tables (`string/number/object-methods`, 698-775) are
  likewise inline;
- the bulk of the shims are in `javatime.janet` (674 lines, ~80% not java.time:
  java.io/util/net/nio/sql/text/math all in one `install-io!`);
- `api.janet:18-56` wires the malli statics + `set-coll-interop!` as load-time
  side effects in the public-API module;
- `core.janet` holds `File`/JDBC constructors; `types.janet` holds the `:jolt/inst`
  representation.

**Target.** A `src/jolt/interop/` directory, one mechanism file + one file per JDK
area, each owning a class's *full* surface (ctor + statics + methods + `instance?`
predicate + canonical name) and exposing one `install!`:

```
src/jolt/interop/
  registry.janet   # MOVED from evaluator: class-statics/tagged-methods/class-ctors,
                   #   register-*!, canonical-names, value-overrides, instance? registry
  coerce.janet     # shared: chr, char->byte, render-piece/writer-piece, pad2,
                   #   the date-format token walker (kills the pad2/render-piece dups)
  java_time.janet  java_io.janet  java_net.janet  java_util.janet
  java_lang.janet  # absorbs evaluator's inline Math/Thread/System/Long/Number +
                   #   string/number/object method tables — the registry becomes the
                   #   ONLY static/method-dispatch mechanism
  jdbc.janet
  install.janet    # (install-all! ctx) — the ONE place api.janet calls (replaces api 18-56)
```

`evaluator.janet`'s `resolve-sym` and the dot-dispatch consult only the registry.
`regex.janet` and `async.janet` stay put — they are engines/library-ports consumed
by interop, not shims.

**Cheap 80% if the full split is too big for one PR:** rename
`javatime.janet`→`host_interop.janet`, pull the four inline static tables + three
method tables out of `evaluator.janet` into it, move `api.janet:18-56` into it,
dedup `pad2`/`chr`/`render-piece`. Collapses the scatter from 5 files to 2.

**Risk:** medium — touches the hot dot-dispatch path; load order must keep the
registry available before any `install!`. Validate with `host-interop-spec` +
the hiccup/markdown/malli example apps + full gate.

---

## Phase 2 — Decompose the god-files

The three biggest interpreter/runtime files are the top LLM-navigability tax.
Split along the cohesive clusters the review mapped (line ranges in the review
notes). Each split is a mechanical move + import; behavior unchanged.

### 2a. `evaluator.janet` (2681 → ~5 files)
- `special_forms.janet` — explode the **680-line `eval-list`** (1921-2599) into
  named `eval-<form>` fns (`eval-fn*`, `eval-let*`, `eval-try`, `eval-dot`, …) +
  a dispatch table. This is the highest-leverage single change: today "where is
  `try` handled" means scanning a 680-line body.
- `resolution.janet` — `resolve-sym`/`resolve-var`/binding/destructuring.
- `ns_loader.janet` — module/bridge plumbing + require/in-ns/use/import/refer.
- `runtime_registration.janet` — protocol/defmulti/deftype/reify setup +
  `install-stateful-fns!` (1506-1909) split into per-domain registration fns.
- (host-interop already left in Phase 1.)
- **De-dup the two `.method` dot arms** (2456-2507 vs 2512-2550): extract one
  `dispatch-member [target field-name args]` used by both. They are copy-pasted
  today and must be hand-synced on every interop change.

### 2b. `core.janet` (3017 → ~6 leaf files)
Clusters are already sequential and mostly leaf — low risk:
`core-types` (predicates/eq/arith/bits), `core-coll` (assoc/seq/transducers/lazy/
transients), `core-print` (pr-str/str), `core-io`, `core-refs` (atoms/vars/delays/
arrays/type), `core-bindings` (the table + `init-core!` + a *labeled* stub
section). The `core-bindings` table stays the single registration point.

### 2c. `passes.clj` (1487 → 3-4 files behind one façade)
The Louvain communities and the cycle analysis agree: the IR-rewriting passes and
the type subsystem are weakly coupled (only `run-passes` + `dirty` shared).
- `jolt/passes.clj` (keep, ~150 lines): `run-passes` + shared state + re-exports —
  the only file the back end imports.
- `jolt/passes/fold.clj` — const-fold (always-on).
- `jolt/passes/inline.clj` — inline + flatten-lets + scalar-replace (share the
  alpha-rename invariant).
- `jolt/passes/types.clj` — type lattice + `infer` + success-checker + driver API
  (kept as one module to respect the inference↔checker cycle; no `declare`
  gymnastics). Extract the `infer` `:invoke` arm (1051-1160) into per-shape
  helpers regardless of the split.
- Fix the **stale ns docstring** (lists 4 passes, omits the type system that is
  >50% of the file).

---

## Phase 3 — Kill the structural duplication

### 3a. One IR op-walk combinator
There are **11 hand-written recursive walks** over the IR op set (const-fold,
inline, subst, body-closed?, pure?, flatten-lets, local-escapes?, subst-lookup,
scalar-replace, infer, backend `emit`). Adding an IR op means editing all of them,
and the "unknown ops pass through" promise is only partly kept. Introduce one
`map-ir-children` (in `ir.clj` or `jolt/ir/walk.clj`) that knows each op's child
positions; rewrite the walks as `(map-ir-children f node)` + their few specials.
Collapses ~400 lines and makes adding an op a one-site change.

### 3b. IR shape hygiene
`:let`/`:loop` bindings are `[name init]` vectors, `:map` pairs are `[k v]`, `:fn`
arities are maps; optional keys are present-or-absent, which is *why* the backend
needs `norm-node`/phm-densification everywhere. Make `ir.clj` constructors always
emit optional keys (nil-valued); delete the defensive `norm-node` calls.

### 3c. Smaller dedups
- `read-delimited` driver for the 4 collection readers (`read-list/vec/set/kvs`)
  in `reader.janet` (the skip/splice logic already drifted once).
- `bucket-index-of` for the 5 stride-2 bucket scans in `phm.janet`.
- Unify the kw-lookup head-matching reimplemented in analyzer/passes/backend
  (a kw-lookup the inference tags but the backend doesn't specialize is a silent
  miss) behind one shared predicate + fn-name table.

---

## Phase 4 — Config & caching coherence

### 4a. Lift run-mode/config resolution into `config.janet`
`config.janet` is 15 lines (one constant); meanwhile `main.janet:585-630` holds
~45 lines of pure env-knob policy (`open-mode?`/`dl`/`optimize?`/shape/whole-program
gates) that can't be unit-tested without the CLI and that the cache keys need to
share. Promote it: `config/resolve-run-mode [argv env] → {:direct-linking? :inline?
:shapes? :map-shapes? :whole-program? :direct-link-auto?}`, plus a canonical
`ctx-shaping-env-vars` list. `main` shrinks to: parse argv → resolve → install →
dispatch.

### 4b. One ctx-image module, two policies
`init-cached` (core image, api.janet) and the deps-image (main.janet) duplicate the
fork→validate→reinstall-print-cb→atomic-publish dance, each with a **hand-built
positional `%q|...` cache key** that silently misaligns if a knob is added.
Extract `ctx_image.janet`: `load-image [path predicate]`, `save-image [ctx path]`,
`ctx-cache-key [pairs]` (derived from `ctx-shaping-env-vars`, impossible to
misalign). Both callers differ only in the validity predicate (source-fingerprint
vs mtime-manifest). This also closes the cache-key footgun for good and resolves
`aot.janet`'s status (fold its marshal helpers in, or delete it — it is exercised
only by one integration test and is on no run path).

---

## Phase 5 — Data structures, reader, types, and the seed↔overlay index

### 5a. `types.janet` (699 → 3-4 files)
Seven concerns share a generic name. Split: `value.janet` (char/inst/uuid),
keep the cohesive `var`+`ns`+`ctx` core, `protocols.janet` (the protocol/type
registry, `type-satisfies?`), `records.janet` (shape-records). Minimum win:
extract the protocol registry + shape-records (the two least "types"-like).

### 5b. `phm.janet` (303 → 3 files)
Split out `lazyseq.janet` (LazySeq has nothing to do with hash maps — pure
mislabel) and `phs.janet` (PersistentHashSet). `phm.janet` keeps the map.

### 5c. Internal collection protocol
phm/pv/plist each re-implement count/seq/conj/predicate/meta with diverging
naming, and `core.janet` dispatches on them with giant per-op `cond`s (every new
structure edits every cond). A minimal `:jolt/type`-keyed vtable (`-count -seq
-conj`) lets core dispatch once. Normalize the trio's naming (`pv` vs `pvec`,
`->`/`-to-`, `EMPTY` vs `EMPTY-PLIST`).

### 5d. Make the seed↔overlay boundary self-documenting
Five fns exist in both the seed (`core-X`) and overlay (`X`) as dispatch twins with
no cross-reference; nothing indexes which copy is authoritative (`transduce` is
overlay-public but `into` is seed-public — surprising and undocumented). Add:
- a generated `REGISTRY` (name → home → public-source → seed-twin? → dispatch-only?)
  with a build-time drift check;
- `SEED-TWIN:` provenance comments on each twin (greppable);
- a distinct prefix for dispatch-only seed helpers so they don't read like public
  ones. Mirrors the existing "delete the seed defn + binding in the same change"
  rule.

### 5e. Boundary doc-comments
Add rep-vs-API pointers between the data structures and `core.janet` (e.g. "the
persistent vector trie lives in `pv.janet`; Clojure-facing vector ops and
tuple/pvec polymorphism live in `core.janet`"), so an agent grepping "vector" in
core knows where the representation is.

---

## Sequencing & rationale

```
Phase 0  (dead code/bugs)         — independent, do first, unblocks clean diffs
Phase 1  (host-interop extract)   — highest value; isolates the recent shim sprawl
Phase 2  (god-file splits)        — biggest navigability win; 2a/2b/2c independent
Phase 3  (op-walk + IR hygiene)   — removes the largest single duplication tax
Phase 4  (config + caching)       — fixes the cache-key footgun; makes boot legible
Phase 5  (data/reader/types + index) — finishes the "one home per feature" goal
```

Phases are independent; within a phase the sub-items (2a/2b/2c, etc.) are separate
PRs. Highest LLM-friendliness per unit risk: **Phase 1**, **2a (`eval-list` split)**,
**3a (op-walk combinator)**, and **5d (seed↔overlay index)**.

Each PR: one concern, full gate green, no behavior change (Phase 0 deletions carry
a regression row).