The reader lowered ^meta on a vector/map/set literal to a runtime
(with-meta form meta) list, so read-string/edn of data with metadata
returned the form and lost the metadata. Attach it to the value instead,
as Clojure does; the analyzer re-emits (with-meta coll meta) for a
meta-carrying collection literal in code, so a literal still carries its
metadata at runtime and ^Type/^long arglist hints (consumed by
analyze-arity directly) are unaffected.
Also: pr honors *print-meta*, and clojure.walk/clojure.edn re-attach
metadata to the collections they rebuild (matches Clojure; a
metadata-driven config lib like aero relies on it).
clojure.test reported a host-condition crash with a blank message (ex-message is
nil for non-ex-info conditions); fall back to jolt.host/condition-message.
condition-message itself left a Chez format-template message (open-input-file's
'failed for ~a: ~(~a~)') unformatted; apply its irritants.
jfile lacked .isAbsolute (path starts with /). clojure.core/default-data-readers
and *data-readers* were nil, so a library merging them (aero's reader opts)
couldn't resolve #inst. Define them keyed by symbol, like Clojure.
(instance? clojure.lang.IObj/IMapEntry/IRecord/IPersistentMap/... x) all returned
false, so libraries branching on these interfaces (e.g. a metadata-preserving
walk guarded by IObj) silently misbehaved. Register an instance-check arm mapping
the interface tokens to jolt predicates, matched by last dotted segment.
(tagged-literal t f) compared by identity, so two equal tagged literals were not
= and didn't dedup as map keys / set members; JVM's TaggedLiteral is value-equal.
Add a register-eq-arm! for jtagged (tag + recursive form); hash is already
structural. Fixes an infinite loop in any fixpoint that dedups structures
containing tagged literals (e.g. aero's #ref expansion).
io/file already resolved a relative path against JOLT_PWD (the user's cwd before
the launcher cd's to the repo root), but slurp and io/reader on a relative string
path opened it against the process cwd — so a program run from its own directory
couldn't read its own relative files (e.g. aero's (read-config "config.edn")).
Route the string branches through project-relative, matching io/file. Internal
callers pass already-resolved paths through read-file-string and are unaffected.
clojure.core/*print-meta* was missing, so (binding [*print-meta* true] …) failed
to compile ("var of non-var"). Add it (default false, like the JVM); corpus
covers binding it.
The EBNF and reader S7 already specified ::kw auto-resolution — the
implementation was out of spec, now aligned. S7 noted unknown ::alias/k MUST be a
read error; jolt is lenient (reads :alias/k), so record that as a deviation.
Corpus gains JVM-certified rows for ::kw resolution, clojure.edn :default
receiving a symbol tag, and with-meta on a lazy seq.
A pushback-reader (drainable, so clojure.edn/read over it works); getLineNumber
is a stub since jolt doesn't track line numbers. Config readers (aero) wrap their
input in one.
A (defmethod m …) where m is :refer'd from another ns passed the bare symbol to
defmethod-setup, which resolved it in the current ns and created a shadow multifn
— the method never reached the real one. Resolve an unqualified name through the
refer table (then current ns) so it lands on the referred multifn.
The AOT build strips the ns form, so the refer table is empty in a binary; emit
chez-register-refer!/-refer-all! per app ns alongside the existing alias
registrations. build-app's fixture gains a defmethod on a referred multifn.
with-meta/meta only recognized the eager collections, so (with-meta (map-indexed
…) m) — a lazy seq — threw "value does not support metadata". A lazy seq is IObj
on the JVM; add it to the metadatable set (keyed in the side-table by identity,
like cseq/procedure).
The reader dropped the namespace on ::kw (read ::foo as :foo), so auto-resolved
keywords never matched their qualified form — code that round-trips them (spec
keys, aero's :aero.core/* expansion keys) silently broke. Resolve ::name against
the current ns and ::alias/name through the alias table, as Clojure does. The
runtime loader reads form-by-form with the ns set after the ns form; the
cross-compile reads all forms up front, so ei-emit-ns*/ei-emit-ns-records set the
ns before reading.
clojure.edn/read over a reader discarded its opts map — :readers/:default/:eof
were ignored, so a custom :default never saw the tag. Route the reader arity
through read-string so opts apply, and pass the tag to :default as a symbol (not
the internal :#name keyword), matching Clojure.
Seed re-minted (the ::halt transducer key in clojure.core now reads as
:clojure.core/halt). Corpus gains ::-keyword rows; the unit case that asserted the
old ns-dropping behavior now asserts the qualified result.
types.clj held the inferencer, the success-type checker, and the driver in one
716-line namespace. Move the self-contained checker into jolt.passes.types.check:
the error-domain predicates (not-number?/not-seqable?/not-callable?), the op
tables, type-name, check-invoke, and the user-fn registry. These are pure over
inferred types and the run's env cells, with no inference, so a check-rule edit
can no longer perturb the inferencer.
The infer-coupled probes stay in types.clj — isolated-diag-count and
check-user-call re-run inference, so moving them would make check depend on the
inferencer and reintroduce the cycle. Verbatim move; new ns wired into
ei-compiler-ns-files; seed re-minted to the byte-fixpoint.
The stderr seam wrote to current-error-port without flushing, so a process that
never returns from -main (a server loop) left its log lines in a buffer that only
drained at exit — they never appeared. Flush each write, like the JVM's
auto-flushing System.err.
The source loader sets the current ns and registers :as aliases per file. The
build flattened every app namespace into one image with no such markers, so all
app forms ran under the last-set ns ("user"). Two breakages followed, both only
in a built binary:
- defmulti/defmethod resolve their target var through chez-current-ns, so they
registered the multifn under "user" while compiled var-derefs used the baked
ns — the multifn the app saw was uninitialized ("not a fn nil" on dispatch).
- a quoted alias-qualified symbol (a (defmethod ig/foo …) on an aliased multifn)
resolves its ns through chez-resolve-alias, but the stripped (ns …) form left
the alias table empty, so it landed in ns "ig".
bld-ns-prelude now emits (set-chez-ns! ns) plus chez-register-alias! for each
ns's :as aliases before that ns's forms, in both the normal and tree-shake emit
paths. The build-app fixture gains a :default multimethod and an aliased cross-ns
defmethod so buildsmoke covers both across all build modes.
walk treated a record as a plain map (record? implies map?), rebuilding it via
(into (empty form) ...) which yields a bare map and drops the type. Add a record
branch before the map branch that conj-es the walked entries back onto the
original, matching JVM clojure.walk's IRecord case. Type-dispatched walks need it
— integrant resolves #ig/ref by detecting its Ref record while postwalking the
config, so without this every ref silently fails to resolve.
clojure.walk is baked into the prelude, so the seed is re-minted. Corpus gains
five JVM-certified rows for record type/instance? survival through pre/postwalk.
infer's :invoke case was ~120 lines of cond arms hand-coding eight call
patterns, all destructured positionally with (nth r 0)/(nth r 1) on the
[type node'] tuples infer returns. Split each pattern into a named helper
(infer-pred-fold/-kw-lookup/-get-lookup/-reduce-hof/-seq-hof/-conj-into/-call)
behind an infer-invoke dispatcher that keeps the cond guards verbatim, and add
ty/nd accessors for the tuple so a silent transposition can't hide.
The accessors are applied only to genuine infer results (the new helpers and
infer-fn-seeded); the :map/:let/:loop arms interleave non-infer pairs
(binding tuples, accumulator pairs) with infer results, so those keep nth.
Pure restructuring — the guards, order, and bodies are unchanged; seed
re-minted to the byte-fixpoint, gate green, 0 new corpus divergences.
The inliner and the type inferencer each recognized (:k m) and (get m :k)
lookups with their own copy of the callee tests — the get-callee check was
duplicated verbatim across both. Lift kw-callee?/get-callee? into
jolt.passes.fold (alongside scalar-const?) and call them from both passes so
the head recognition can't drift.
Only the head predicates move. The deliberate differences stay: the inliner
still accepts any scalar key in the get-form (its scalar-replacement targets
can be string/number-keyed maps) while the inferencer requires keyword keys
for struct field typing, and the inferencer keeps its two arms separate so each
rebuilds args for its form. The backend's value-as-fn ifn-kind is left alone.
numeric.clj dbl-spec/lng-spec and backend_scheme.clj dbl-ops/lng-ops must agree —
a spec'd op with no table entry makes emit-numeric splice a nil op string. Document
the contract on both sides. Comment-only; seed re-mints byte-identical, gate green.
The other two ideas in this bead were rejected after inspecting the code:
- collapsing inline.clj local-escapes? onto reduce-ir-children would reintroduce the
under-reporting hazard its docstring deliberately guards against (default-true is
load-bearing for scalar-replacement soundness).
- folding numeric recur-kinds/recur-arg-lists into one walk loses the type-env
threading recur-kinds needs through :let; the parallel split is justified.
analyzer.clj referred jolt.host/form-char? but never called it (form-char? stays
live — backend_scheme.clj uses it). Promote numeric.clj an-invoke's :wild operand
rule (an integer literal is valid in either fl/fx kind) from an inline comment to the
function docstring. Both output-neutral: the seed re-mints byte-identical, gate green.
Two small clarity moves from the review:
- dynamic-vars.ss -> dynamic-var-defaults.ss. It holds the default VALUES of a few
core dynamic vars (*clojure-version*, *assert*, …); the near-identical dyn-binding.ss
holds the binding-stack machinery. The names were easy to confuse.
- Pull the ~60-line %-format engine out of the natives-misc.ss grab-bag into
natives-format.ss (its ->long/pad-left/fmt-float helpers were local to it).
rt.ss loads + MODULES.md updated. Runtime .ss, no re-mint; make test green, format +
the dynamic vars verified.
natives-parity.ss was a grab-bag (its own header called it 'parity'). Dissolve it
into homes that say what they hold:
- cat -> natives-transduce.ss (was natives-xform.ss, renamed for the
whole transducer surface: volatiles + cat + transduce/sequence)
- transient? -> transients.ss
- rseq -> natives-seq.ss
- hash family -> natives-misc.ss (the public hash API over jolt-hash)
The remainder — the #?() feature set, the reader-conditional + re-matcher tagged-map
ctors, and macroexpand — is a coherent reader/macro runtime-support unit, kept as
natives-reader.ss (np- prefix -> nr-). rt.ss loads + two comment refs updated.
Runtime .ss, no re-mint. make test green; hash/transient?/rseq/cat/macroexpand and
#?() reader features all verified.
The trio split on a fine axis (registry core / statics / object classes) but the
names didn't say so — 'static-statics'/'static-objects' and headers that read
'Continues X'. Rename:
host-static-statics.ss -> host-static-methods.ss (Class/member statics + fields)
host-static-objects.ss -> host-static-classes.ss (instantiable object classes)
host-static.ss stays the registry core. Headers rewritten to state each file's role
and what it covers instead of chaining. rt.ss loads + the one comment reference +
MODULES.md updated. No code moved; runtime .ss, make test green.
Under --direct-link a top-level def binds jv$<fqn> and app->app calls bound directly
to it. emit-invoke raw-applied that binding for any var callee, but only a fn-valued
def is a Scheme procedure: (def cfg {...}) then (cfg :a) emitted (jv$cfg :a), applying
a pmap -> "attempt to apply non-procedure". Maps/sets/keywords are invokable in Clojure
via jolt-invoke, which the indirect path used, so this only bit closed-world builds.
Track which direct-linked vars hold fn literals (direct-link-fns, registered at the def
site when the init op is :fn) and only raw-apply those. A non-fn callee falls through to
the jolt-invoke branch, which still uses the direct jv$ binding as the invoke target —
so the var-deref is still skipped, just not the dispatch.
Seed source: re-minted. Regression in directlink-test.ss (jolt-cw1o).
The printer's two entry points (jolt-pr-str in rt.ss, jolt-pr-readable in printing.ss)
get register-pr-str-arm! / register-pr-readable-arm!, plus register-pr-arm! for the
types whose str and readable forms match (bigdec/inst/uuid/tagged/record/ns/var). The
normalize arms (sorted, lazy-seq, queue) and the uri readable arm register per-printer.
Also folds in the hash (dyn-binding var-cell), class (io uri/uuid/file), and get
(transients) arms missed earlier.
natives-array's get stays a case-lambda wrapper on purpose: its 2-arg path errors on
an out-of-bounds index while the 3-arg path returns the default, an arity distinction
the (coll k d) registry collapses — left as-is to preserve behaviour.
Completes jolt-lmot: all six dispatchers (hash/class/get/=/pr-str/pr-readable) off the
set!-rebind chains. make test green, 0 new corpus divergences; pr-str/str of inst,
uuid, bigdec, sorted-map, record-with-lazyseq, queue all verified.
jolt-get (4 sites: host-table/natives-misc/inst-time/records) and jolt=2 (6 sites:
records/vars/inst-time/natives-misc/bigdec/host-table) move off the set!-rebind
chains to register-get-arm! / register-eq-arm!. get's case-lambda becomes a stable
2/3-arg entry over a 3-arg dispatch; the equality arm pred is (a b) since either arg
may carry the type, and the host-table sorted arm normalizes-then-re-dispatches.
Behaviour-preserving (runtime .ss): var/inst/bigdec/record/uuid equality, record!=map,
sorted-map=plain-map, and all the get cases verified; make test green, 0 new corpus
divergences. Four of six dispatchers done; the printer (pr-str/pr-readable) remains.
Replace the set!-capture-and-rebind chains extending jolt-hash (3 sites: inst-time/
host-table/records) and jolt-class (3 sites: bigdec/natives-queue/host-table) with a
register-hash-arm! / register-class-arm! registry (the pattern already used by
register-str-render!). The base dispatcher walks its arms — disjoint types, so order
is immaterial — then falls to the base cases. The entry is stable, so the per-site
(def-var! "clojure.core" "class" …) re-points are gone.
Behaviour-preserving (runtime .ss, no re-mint): jinst hash, bigdec/queue/record
class, record hash, and a sorted-map hashing as its plain map all verified; make test
green, 0 new corpus divergences. First two of six dispatchers; get/=/pr-str follow.
set-optimize!/set-direct-link! are process-global flags in the back end, set then
reset around the emit. A strict-form build error (failing forms error the build by
design) skipped the reset, leaving the compiler in optimize/direct-link mode for any
later in-process caller. dynamic-wind guarantees the revert. Behaviour unchanged on
the success path; both --tree-shake and --opt --direct-link build + run identically.
Tree-shaking was split across emit-image.ss (the dce-* helpers + record producer)
and build.ss (bld-shake-all + the manifest splice), with the DCE record accessed by
raw (vector-ref r 0..3) in ~10 places and the manifest splice/drop driven by
substring-matching (load "…prelude.ss").
- New host/chez/dce.ss owns the whole DCE concern: a named record API
(dce-rec/-keep?/-fqn/-refs/-str — no more positional vector indexing), the ref
extraction + ref-set constants, dce-blob-records, and dce-shake decomposed into
dce-build-graph / dce-reachable / dce-bail-scan / dce-partition (was one 50-line
bld-shake-all doing five jobs with shared mutable state).
- emit-image.ss keeps only ei-emit-ns-records (it drives the ei-* compiler) and uses
the dce-rec constructor.
- The runtime manifest is now tagged ('prelude/'image/'compile-eval); bld-emit-runtime
dispatches on the tag instead of substring-matching file paths, so the core-splice
and compiler-drop can't silently break on a rename/reorder.
Behaviour-preserving (runtime .ss, no re-mint): build-app shakes identically
(56/460, 8.12MB), make test green, make shakesmoke green (4 git-lib apps
byte-identical, same sizes).
Navigability groundwork from the architecture review — zero behaviour change.
- docs/MODULES.md: the repo map. Area -> directory -> key files -> re-mint?, plus
per-feature touch points (tree-shaking, direct-linking, numeric fl/fx, inlining,
multimethods, deps) and where a given clojure.core fn lives. Answers "where does
X live / what's related to Y" in one read.
- docs/rfc/README.md: index the 7 RFCs; flags RFC 0007's stale "no code yet" status
(direct-linking + tree-shaking shipped) and the undocumented inlining/numeric work.
- CLAUDE.md: document the var-deref calling convention (public defns reached from the
.ss runtime by string lookup aren't dead), the def-var! native pattern, and the
overlay shadowing rule; point at MODULES.md.
- REFACTOR_PLAN.md: the prioritized, risk-tiered plan (working doc for this branch).
README + tools-deps.md cover --tree-shake and --direct-link: what tree-shaking does
(whole-program reachability over app + libraries + clojure.core, drop unreachable,
drop the compiler for no-eval apps), and why it bails to keep-all when reachable code
resolves vars by name at runtime (eval/resolve/ns-resolve/...), with the diagnostic
output and how to make an app shakeable. Notes the Stalin soundness model.
When --tree-shake keeps everything because reachable code resolves vars at runtime,
list the offending def -> bail-ref pairs (up to 6) instead of just saying it
skipped. Makes it actionable: e.g. ring-app shows
clojure.tools.logging/call-str -> ns-resolve and selmer.filters/generate-json ->
resolve, so you know which library (not your code) blocks shaking.
--tree-shake now shakes the clojure.core/stdlib prelude in the same reachability
graph as the app + libraries — only core fns actually reached from -main ship.
dce-blob-records reads prelude.ss with Chez `read`, unwraps each
(guard ... (def-var! "ns" "name" V)) and builds the core->core call graph from the
(var-deref/jolt-var "ns" "name") refs in V — the real emitted edges, no
re-analysis. bld-shake-all merges core + app records into one BFS; roots are -main,
side-effecting forms, and the clojure.core fns the runtime .ss shims reference by
name (enumerated in dce-runtime-core-roots). The shaken core is spliced where the
prelude.ss blob was, in original (tier) order, so load-time deps are preserved.
Bail (reachable runtime resolve) keeps the prelude whole.
Soundness follows Stalin's rule (any reference, value or call, keeps a def live):
dce-collect-refs counts :var and :the-var; non-def registration forms are always
kept (covers protocol/multimethod dispatch). Validated by make shakesmoke: the four
deps.edn git-lib apps build byte-identical output shaken vs not.
Wins (binary): build-app 9.84MB -> 8.12MB (dropped 403/457 core defs); malli-app
10.0MB -> 8.1MB; markdown 9.9 -> 8.3; commonmark 9.8 -> 8.1 — all output-identical.
build-smoke asserts an unused core fn (group-by) is dropped; full make test green.
Two things, both from studying Stalin's closed-world DCE:
1. Soundness fix: dce-collect-refs now counts a :the-var (#'x / (var x)) reference,
not just a :var. Stalin's rule is that ANY reference — value position, not only a
direct call — keeps the target live; a var referenced as #'x would otherwise be
wrongly dropped. (My :var collection already covered value-position refs; this
closes the the-var hole.)
2. host/chez/tree-shake-smoke.sh (make shakesmoke): builds example apps default vs
--tree-shake and requires identical output — the real risk is shaking a binary
that pulled libraries via deps.edn. Covers markdown/malli/commonmark/hiccup
(git-lib apps). All produce byte-identical output shaken vs not, and drop
~0.8-1MB (malli 10.0MB -> 9.0MB) from the compiler-drop. Slow; not in the default
gate. Skips without the examples repo.
An AOT-compiled app only needs the analyzer/back end at runtime to compile from
source — eval / load-string / load-file. Macros are expanded at build time and a
require of a baked namespace no-ops, so a closed app that never compiles at runtime
carries the compiler image (~0.8MB) as dead weight.
Under --tree-shake, when reachability is trustworthy (no bail) and no reachable code
references eval/load-string/load-file/load-reader/load, omit host/chez/seed/image.ss
+ compile-eval.ss from the runtime manifest. bld-tree-shake returns the flag
alongside the shaken forms; bld-emit-runtime filters the manifest.
Measured: build-app 9.84MB -> 9.05MB, still runs. Safety verified: an app that evals
keeps the compiler (eval is a bail + compile ref) and eval works at runtime.
build-smoke asserts the compiler is gone in the no-eval app; full make test green.
`jolt build --tree-shake` (or deps.edn :jolt/build {:tree-shake true}) does
reachability DCE over the re-emitted app + library namespaces: keep -main, every
side-effecting (non-def) top-level form, and every def reachable from those; drop
the rest. A macro (expanded at AOT, never called at runtime) is prunable too.
Sound: bails (keeps everything) if REACHABLE code resolves vars by name at runtime
(eval/resolve/ns-resolve/requiring-resolve/find-var/intern/load-string/...), which a
static call graph can't follow. Unreached eval-using library code is simply shaken
away and never triggers the bail. clojure.core and the compiler image stay baked
(prelude + image blobs), so only re-emitted namespaces are shaken for now.
The reachability machinery is in emit-image.ss (records: keep?/fqn/refs/str via
reduce-ir-children) + build.ss (BFS + bail check). build-smoke covers it (drops the
unreachable `twice` macro, output unchanged). Opt-in; default builds are untouched.
full make test green.
Scope note: this shakes the re-emitted app/lib code only. Measurement shows jolt's
compiled code is ~5.8MB of a ~9.8MB binary, dominated by the clojure.core prelude
(~1.5-2MB) and the compiler image (~0.8MB) — both baked blobs this pass doesn't
touch. Those (shake-core, drop-compiler-when-no-eval) are the larger footprint wins,
filed as follow-ups.
A loop var with an integer-literal init now types :long (fx ops) when every recur
arg in its slot is an increment-style step — the var unchanged, inc/dec, or (+/-
var <int-literal>). So (loop [i 0] (recur (inc i))) gets fx1+/fx<? without a hint,
matching how Clojure treats a primitive-long loop counter.
Soundness: only increment steps qualify. A multiplicative or large-growth
accumulator like (recur (* acc 2)) is never seeded, so it stays generic and keeps
arbitrary precision — a bignum-producing loop (e.g. a factorial) is unaffected.
counter-step? gates this; the existing fixpoint demotes anything inconsistent.
test/chez/numeric-test.ss 44/44 (incl. a factorial loop staying bignum-exact while
its counter is fx); full make test green, 0 new corpus divergences.
