An AOT-compiled app only needs the analyzer/back end at runtime to compile from
source — eval / load-string / load-file. Macros are expanded at build time and a
require of a baked namespace no-ops, so a closed app that never compiles at runtime
carries the compiler image (~0.8MB) as dead weight.
Under --tree-shake, when reachability is trustworthy (no bail) and no reachable code
references eval/load-string/load-file/load-reader/load, omit host/chez/seed/image.ss
+ compile-eval.ss from the runtime manifest. bld-tree-shake returns the flag
alongside the shaken forms; bld-emit-runtime filters the manifest.
Measured: build-app 9.84MB -> 9.05MB, still runs. Safety verified: an app that evals
keeps the compiler (eval is a bail + compile ref) and eval works at runtime.
build-smoke asserts the compiler is gone in the no-eval app; full make test green.
`jolt build --tree-shake` (or deps.edn :jolt/build {:tree-shake true}) does
reachability DCE over the re-emitted app + library namespaces: keep -main, every
side-effecting (non-def) top-level form, and every def reachable from those; drop
the rest. A macro (expanded at AOT, never called at runtime) is prunable too.
Sound: bails (keeps everything) if REACHABLE code resolves vars by name at runtime
(eval/resolve/ns-resolve/requiring-resolve/find-var/intern/load-string/...), which a
static call graph can't follow. Unreached eval-using library code is simply shaken
away and never triggers the bail. clojure.core and the compiler image stay baked
(prelude + image blobs), so only re-emitted namespaces are shaken for now.
The reachability machinery is in emit-image.ss (records: keep?/fqn/refs/str via
reduce-ir-children) + build.ss (BFS + bail check). build-smoke covers it (drops the
unreachable `twice` macro, output unchanged). Opt-in; default builds are untouched.
full make test green.
Scope note: this shakes the re-emitted app/lib code only. Measurement shows jolt's
compiled code is ~5.8MB of a ~9.8MB binary, dominated by the clojure.core prelude
(~1.5-2MB) and the compiler image (~0.8MB) — both baked blobs this pass doesn't
touch. Those (shake-core, drop-compiler-when-no-eval) are the larger footprint wins,
filed as follow-ups.
A loop var with an integer-literal init now types :long (fx ops) when every recur
arg in its slot is an increment-style step — the var unchanged, inc/dec, or (+/-
var <int-literal>). So (loop [i 0] (recur (inc i))) gets fx1+/fx<? without a hint,
matching how Clojure treats a primitive-long loop counter.
Soundness: only increment steps qualify. A multiplicative or large-growth
accumulator like (recur (* acc 2)) is never seeded, so it stays generic and keeps
arbitrary precision — a bignum-producing loop (e.g. a factorial) is unaffected.
counter-step? gates this; the existing fixpoint demotes anything inconsistent.
test/chez/numeric-test.ss 44/44 (incl. a factorial loop staying bignum-exact while
its counter is fx); full make test green, 0 new corpus divergences.
jolt.passes.inline was fully written but dormant — it fetched bodies via the
inline-ir host hook, which was a stub returning nil. Wire it up: run-passes stashes
each inline-eligible defn (single fixed arity) as its form is optimized, and
inline-ir hands the body back at call sites under --opt.
The catch was the ^double/^long coercion: an inlined fn drops its param-entry and
return coercion, so (work 3 4) on a ^double fn would return 25 instead of 25.0. New
:coerce IR node carries the coercion inside the spliced body — the inline pass wraps
a hinted param's arg and the return in :coerce, the back end lowers it
(exact->inexact / jolt->fx), and jolt.passes.numeric reads its :kind. So an inlined
call matches the called one and the body's fl*/fx* fast path still fires.
Only under --opt (closed world); the seed mint and -e don't inline, so selfhost and
the corpus are unaffected. test/chez/inline-test.ss 12/12 (make inline); full make
test green, 0 new corpus divergences.
Bench (hot loop, body is a ^double helper call): direct-link 500ms -> --opt
(inlined) 184ms = 2.7x, by eliminating the call + coercion wrappers and letting Chez
fuse the fl-ops unboxed. ~26x over the default dispatched build.
A ^double/^long return hint on a fn's name now (a) coerces the fn's value on the
way out — exact->inexact / jolt->fx, like a JVM primitive return — and (b) types a
call to it, so an accumulator over the result specializes:
(defn ^double work [^double x ^double y] (+ (* x x) (* y y)))
(loop [acc 0.0] (recur (+ acc (work a b)))) ; (+ acc (work ..)) -> fl+
The analyzer pushes the name's numeric tag onto each arity (:ret-nhint) for the
back-end coercion, and resolve-global surfaces the callee's declared return
(:num-ret, read from var meta) onto the :var node so jolt.passes.numeric types the
call. defn carries the name hint through.
This unblocks the accumulator-over-fn-result pattern that round 2 had to demote.
The win is bounded by call overhead in an open/dispatched build (~1.15x on a hot
loop whose body is a helper call); it compounds with direct-linking and, later,
inlining. A numeric return hint is a contract, like ^long — redefining the var to
return another type in an open build breaks it.
Not yet: per-arity arglist return hints, (defn f (^double [x] ..)). Gate:
test/chez/numeric-test.ss 39/39; full make test green, 0 new corpus divergences.
loop-kinds only typed :double accumulators; a ^long-seeded loop var (e.g.
(loop [acc start] ...) with a ^long start) stayed generic even though it's sound
to fx-type — :long only ever comes from an explicit hint, and a ^long value is
already coerced to a fixnum at fn entry. Keep the init's kind (:double or :long)
through the fixpoint, demoting only on a recur-arg mismatch.
Integer-literal-init loop vars (a bare (loop [i 0] ...)) still stay generic by
design: :long is never seeded from a literal, so a bignum-producing loop keeps
arbitrary precision.
A loop binding whose init is double and whose every recur arg stays double (a
bounded monotone fixpoint) is typed :double, so its arithmetic — and the recur
args feeding it — emit fl-ops. Chez can then keep the accumulator unboxed in a
float register across the loop.
Integer loop vars stay untyped: a bare integer init never seeds :long (same rule
as round 1), so a bignum-producing loop keeps arbitrary precision rather than
overflowing a fixnum. recur-kinds walks only tail position (if/do-ret/let-body),
stopping at nested loop/fn so a loop sees only its own recur.
A/B on a loop-carried double accumulator: 735ms generic -> 500ms typed (1.47x),
closing the gap to the JVM from ~3.3x to ~2.2x. The integer counter stays generic,
which is most of the residual.
A ^double/^long param hint (or a float literal) now drives Chez flonum/fixnum
ops instead of generic arithmetic — JVM-style primitive hints, available in every
build and at -e (not gated on direct-linking or whole-program inference).
New pass jolt.passes.numeric: a local forward type-flow seeded from ^double/^long
fn-param hints (analyzer attaches :nhints per arity) and float literals,
propagated through let inits / arithmetic / if / do. It tags an arithmetic invoke
:num-kind :double|:long when every operand is that kind (an integer literal is a
wildcard, coerced to a flonum in a double op). The back end lowers a tagged node
to fl+/fl-/fl*/fl//fl<?/... or fx+/fx*/fx1+/fxquotient/... (unchecked-add etc.
join the fixnum path; == too). Runs last in run-passes, both branches.
Soundness: :long is seeded ONLY from an explicit ^long hint, never a bare integer
literal, so un-hinted integer code keeps jolt's arbitrary-precision numbers — no
fixnum-overflow surprise, no corpus divergence. :double comes from ^double hints
and float literals (flonum arithmetic is always flonum, matching the generic
result). A ^long hint is a promise the value is a fixnum: fx+ raises on overflow,
like a JVM fixed-width long.
Numeric-hinted params coerce at fn entry (exact->inexact / jolt->fx), the way the
JVM coerces a primitive parameter — so the body's fl*/fx* ops can rely on the
type even when a caller passes an exact int (e.g. Chez's (* 0 1.0) => exact 0).
Round 1 specializes hinted straight-line / fn-body arithmetic. fl-ops are ~4x
generic in a tight Chez loop, but realizing that on loop-carried accumulators
needs loop-var typing — round 2. Sound foundation, gated by test/chez/numeric-test.ss.
Release builds can legitimately want runtime dynamism (redefinition, eval,
load-string), so closed-world direct-linking shouldn't be forced on them. Gate it
behind an explicit --direct-link flag (or deps.edn :jolt/build {:direct-link
true}); off by default in every mode, including release and --opt.
build-binary takes an explicit direct-link? arg instead of deriving it from the
mode. build-smoke now covers the --direct-link path and asserts the cross-ns call
actually lowers to a jv$ binding; default release stays dynamically linked.
A release/optimized `jolt build` is a closed world: every app def is final, so
an app->app call can bind to the def's Scheme binding directly instead of going
through (jolt-invoke (var-deref ns name)).
The emitter gains a direct-link mode (off for the seed mint, runtime -e/repl, and
dev builds). With it on, a top-level app def also emits a binding jv$<ns>$<name>
that def-var! aliases; an app->app call or value-ref to a name already emitted in
the unit lowers to that binding, skipping both the var-table lookup and the
generic IFn dispatch. ^:dynamic/^:redef defs and nested defs (a defonce's inner
def) opt out and stay indirect. Off direct-link mode, emit-top-form is exactly
emit, so the seed and runtime eval are byte-unchanged (selfhost holds).
build.ss turns it on for release + optimized; the defined-set accumulates across
the dependency-ordered namespaces so a dep's defs are linkable by the time the
entry that calls them is emitted. App->core calls stay indirect for now (core is
the baked seed); that's a later stage.
~1.74x on a hot cross-namespace call loop (26.5s -> 15.2s).
Build output landed in the CLI's cwd (the jolt repo, since bin/joltc cd's
there), not the project — so a bare -o path or the default binary appeared
in the wrong place. Resolve output against JOLT_PWD, and default it cargo-
style under the project's target/: target/release for release/--opt,
target/debug for --dev, named after the project dir. The <name>.build scratch
dir sits beside the binary, so it lands under the same target dir. -o is
honored — absolute as-is, relative against the project.
A built binary dropped its deps.edn :jolt/native declarations and its
resource roots, so an FFI+resources app (ring-app) failed at runtime:
sockets/sqlite gave 'no entry for socket' and io/resource returned nil.
The buildsmoke fixture is pure compute, so neither path was exercised.
The launcher now loads required + :process native libs before the app's
top-level forms (a library's defcfn resolves its foreign-procedure symbols
at top-level eval during startup, so the libs must be loaded first);
optional libs load in the scheme-start launcher, where a missing lib is
caught rather than aborting the heap build.
deps.edn :jolt/build {:embed [dirs]} bakes those dirs' files into the heap
(register-embedded-resource! at heap build), so io/resource serves them with
no files on disk. Non-embedded resources resolve at runtime against JOLT_PWD,
and io/file reads (e.g. config.edn) stay external.
build-binary now takes the encoded natives, embed dirs, and project paths
from cmd-build; deps/resolve-project surfaces them. Buildsmoke fixture grows
an embedded resource + a :process native to cover both paths.
jolt could call C (foreign-fn -> foreign-procedure) but C could not call back
into jolt, which GTK signals (and any callback-taking C API) require. Add the
inverse: jolt.ffi/foreign-callable wraps a jolt fn as a C-callable function
pointer, mirroring the foreign-fn pipeline.
A new jolt.ffi/__ccallable special form carries the fn as a child expression
(analyzed + walked by the passes; ir.clj gains an :ffi-callable arm in both
child walks) plus literal arg/ret type keywords. The back end lowers it to a
locked Chez foreign-callable and returns its entry-point address as a jolt
pointer; host/chez/ffi.ss registers the code object so the collector keeps it,
and free-callable unlocks it. :collect-safe emits the convention that
reactivates the thread on entry, for callbacks fired while it is parked in a
:blocking call (a GTK main loop).
Test: ffi-binding-test.ss sorts an int array through libc qsort with a jolt
comparator (C -> jolt -> C). Re-minted seed.
Dead code removal, O(n^2) hot-path fixes (transients/queues/deps), wiring the
optimization pass pipeline into compile + build, deterministic seed emission,
splitting the oversized files (20-coll, host-static, records, types lattice),
and restructuring the two worst maintainability smells: the str-render /
instance-check set!-override chains became registries, and the type-inference
walk now threads an immutable env instead of ~14 module atoms.
Adds an inference gate (make infer, 26 cases) so the type pass — which the
corpus/unit gates don't exercise — has real coverage. Full gate green:
self-host fixpoint holds, corpus 2735 (0 new divergences).
Adds 3 cases for the opt-path checker (set-check-mode! -> run-inference ->
take-diags!): diagnostics drain once, the buffer empties on re-drain, and
check-mode off produces none. This is the public checker entry the other 23
cases didn't touch. Runtime-only.
types.clj drove inference through ~14 module-level atoms; the infer walk was
non-reentrant and depended on hidden set-*! install order. Thread one immutable
env (mk-env) through infer instead: it snapshots the installed config
(rtenv/vtypes/record-shapes/protocol-methods/map-shapes?) and carries the
per-run flags and accumulator/guard cells (diags/calls/checking-set/diag-memo).
A fresh env per run makes the pass re-entrant — isolated-diag-count's probe now
runs under a sub-env with its own diags cell instead of save/restoring a shared
atom.
Only state whose lifecycle spans separate API calls stays module-level: a
config-box the set-*! API writes, the escapes/user-sig sweep registries, and a
bridge holding the last checking run's diags for take-diags!. record-type-from-
entry/field-type-from-tag now take the shapes map directly rather than reading a
global.
jolt-ogib.10. Behavior pinned by the new infer gate (23 cases) plus selfhost +
buildsmoke. Re-minted seed.
The corpus/unit gates compile through run-passes' const-fold-only branch, so
the type-inference walk runs only under jolt build --opt — buildsmoke hit one
trivial app and checked stdout. run-infer.ss drives the pass directly: analyze
a source string, then call check-form / infer-body / the set-*! registries and
assert diagnostic counts and collected calls/escapes. Wired into make ci.
Gives the inference pass real behavior coverage so refactoring its internal
state is gate-validatable. jolt-ogib.10 groundwork.
jolt-str-render-one and instance-check were each extended by a chain of
set!-wrapping closures spread across ~10 and ~5 host files, so the real
behavior of either was scattered and load-order-dependent. Give each a
registry the base file owns: converters.ss/records-interop.ss define the
registry plus a register-* helper, and each extending file registers one arm
instead of capturing %prev and set!-ing the global.
str-render arms are type-disjoint; instance-check arms run newest-first (the
old outermost-wins order) and may return 'pass to defer. The string-token ->
symbol normalization the natives-array arm did for every inner arm moves to
the dispatcher head; array tokens stay strings for that arm to decide.
jolt-ogib.14. Runtime-only shims, no re-mint.
The lattice-split commit staged its seed before make remint ran, so image.ss
lagged the source. Commit the correct re-minted image (gensym renumbering only).
types.clj was 852 lines mixing the pure structural-type algebra with the
inference engine, checker, and driver. Move the lattice — scalar/struct/vec/set/
union types, join-t, depth-cap, shape, and the numeric/vector return-fn sets —
into jolt.passes.types.lattice (no inference state, no requires). types.clj
requires it; the engine is now ~720 lines. Compiled into the image before
jolt.passes.types. Re-minted seed differs only by gensym label renumbering.
records.ss mixed the record/protocol/reify model with JVM exception emulation.
Move the contiguous ex-info accessors, the exception supertype hierarchy,
instance-check, case-string, and the instance-check def-var into records-interop.ss,
loaded right after records.ss. Those are only called at runtime, so the relocated
forms resolve fine; records.ss is now the value model and protocol dispatch.
Runtime shim — no seed change.
The 929-line interop registry split three ways: host-static.ss keeps the
registries, the jhost record, the emit entry points and coercion helpers;
host-static-statics.ss holds the java.lang/util static-method registrations;
host-static-objects.ss holds the host object classes (ArrayList, HashMap, the
reader/writer/tokenizer shims, ctors, URL codecs) and the instance? hook. rt.ss
loads the three in order. Runtime shim — no seed change.
The 1123-line collection tier is the largest source file. Cut it at two existing
section banners into 20-coll (predicates, printing, hierarchies, pure-over-core
leaves), 21-coll (rand/sort seams, the test runner, fn combinators), and 22-coll
(canonical Clojure ports, transduce/into, JVM-shape stubs). No macros in this tier,
so order is the only constraint; the emit-image manifest lists the three in
sequence. Re-minted seed is identical apart from gensym label renumbering.
check-user-call rebuilt the all-:any env once per parameter (O(params^2)) and
re-inferred a callee body at every call site. Build the env once and memoize each
probe by [key i argtype] (and the baseline by [:base key]), cleared per form in
check-form. The global type-env is stable within a form's check and the probe's
calls/escapes side effects aren't read there, so a skipped repeat is observably
identical. (The inline-side re-walk the audit flagged is moot: hc-inline-ir is a
no-op on Chez, so try-inline never reaches body-size/body-closed?.)
ei-emit-ns (emit-image) and bld-emit-ns (build) were near-verbatim copies that had
drifted: the minter guard-wraps and skips failing forms, the build is strict, and
since the passes were wired the build also runs run-passes. Fold both into
ei-emit-ns* with optimize?/guard? flags; ei-emit-ns and bld-emit-ns become one-line
callers. Output is byte-identical (selfhost fixpoint and build smoke stay green).
- str-join delegates to str-join-strs (it only adds the per-element render).
- loader: extract resolve-on-roots; find-ns-file and load both use it.
- NumberFormat registers its short + FQ names from one shared member list.
- inst-time's private floor-div/floor-mod renamed inst-floor-* so they don't read
as the math.ss reals version.
Left the fold/inline/types pure-fn sets and the keyword/symbol ns builders alone:
those are file-local and semantically distinct (e.g. the intern key uses NUL, not
"/"), so merging them would be wrong.
- take-last / drop-last return seqs, not vectors: take-last wraps in seq; drop-last
is the JVM (map (fn [x _] x) coll (drop n coll)) form (lazy, () when empty).
- cycle is lazy ((lazy-seq (concat coll (cycle coll)))) so it no longer counts its
argument and terminates on a lazy/infinite input.
- fold's foldable-call catch uses :default, matching the rest of jolt-core and
also catching a raw host condition from a folding primitive.
- alts! rejects non-channel ports with a clear error (put specs / :default are
unsupported) instead of crashing inside ac-poll!.
- Misc: drop the unreachable second getCause clause; jolt-nth on a string raises
'nth "index out of bounds" like the vector branch; name the inline fixpoint cap;
bld-sh-capture rejoins lines with newlines; clarify a couple of comments.
map-ir-children single-sourced the child layout for rewrite passes; the read-only
analyses each re-enumerated ops by hand. Add a fold companion, reduce-ir-children,
and rebuild body-size, pure?, and body-closed? on it (each reduces to a leaf value
+ the special ops it actually needs). local-escapes? stays an explicit walk — its
default is conservatively true and it inspects node shape beyond child purity, so
folding an unhandled op over its children would be unsound for scalar replacement.
analyze-special inlined def (~35 lines) and set! while try/letfn/fn* were already
helpers. Pull both out and move field-head? above analyze-special so its set! arm
and analyze-list reach it without a forward reference — the file's "only analyze
is forward-declared" invariant holds again. Pure code motion.
The fold/inline/types passes and the jolt.passes façade were baked into neither
seed half and never invoked: compile-eval and build went analyze -> emit directly,
and `jolt build --opt` flipped an optimize flag that nothing consumed.
- Compile the passes into the image (emit-image manifest): fold, inline, types,
then the jolt.passes façade, after jolt.ir.
- compile-eval and build.ss now run jolt.passes/run-passes between analyze and
emit. Off the direct-link path it is a pure const-fold; `jolt build --opt`
turns on inline + flatten + scalar-replace + type inference (it sets
hc-optimize?, which inline-enabled? reads).
- The seed minter (emit-image) stays analyze -> emit, so the seed is built
un-optimized and the self-host fixpoint is unaffected.
build-smoke already exercised --opt; it now actually optimizes and still matches
the release binary's output. Corpus floor and the fixpoint are green.
Round 1 (correctness + dead code):
- Fix duplicate java.util.HashMap registration in host-static.ss: the alist
impl shadowed the hashtable ctor while leaving the hashtable methods bound,
so .keySet/.values/.remove/.clear crashed. Drop the alist version.
- Delete jolt-core/jolt/reader.clj: a 463-line dead duplicate reader, never
required or compiled (the live reader is host/chez/reader.ss) and drifted.
- Remove dead defs: ir/rt + :rt op + unused ir/op; the Janet branch in
clojure.edn/drain-reader; a shadowed first clojure.string/trim-newline;
io.ss jolt-char-array + the reader def-var (both shadowed by natives-array);
concurrency.ss jolt-future-done?*; compile-eval.ss jolt-analyze-emit.
Round 2 (perf + determinism):
- emit-quoted-map-value / quoted sets now emit sorted by emitted text instead
of host-hash order, which isn't stable across Chez versions (jolt-8479).
- jolt-into folds through a transient, so into/vec/mapv/filterv onto a vector
are O(n) instead of O(n^2).
- deps resolve-deps walks its queue with an index cursor (was subvec-per-pop).
- async channel and agent action queues use amortized-O(1) FIFOs; ArrayList is
backed by a growable vector (O(1) add/get) instead of a list.
Thread/yield was a no-op and Thread/interrupted always returned false. Now:
- yield calls libc sched_yield (resolved once via the process symbols), so a
spin loop relinquishes the CPU. Falls back to a zero-length park if the symbol
can't be resolved.
- each OS thread carries an interrupt flag (a box, thread-local). currentThread
returns a handle wrapping the calling thread's flag, so .interrupt from another
thread sets the target's flag. .isInterrupted reads without clearing; the static
Thread/interrupted reads and clears — JVM semantics.
Consolidates the Thread surface: currentThread + the instance methods live in
io.ss (where the handle and its classloader are built), the flag box + yield +
the interrupted static in host-static.ss. Unit cases cover yield, the read/clear
split, and a cross-thread interrupt over a future.
The from-source Chez build failed on expeditor.c needing X11/Xlib.h — the
expression editor's clipboard. Configure with --disable-x11 (not needed in CI)
and bump the cache key. Add a "Compile a binary" section to the README.
The apt chezscheme package ships petite+scheme only — no kernel dev files — so
the standalone-binary gate skipped on CI, leaving the whole jolt build pipeline
and the --opt inference passes uncovered on Linux. Build Chez v10.4.1 from
source (cached) to get libkernel.a + scheme.h, install the libs the kernel links
against, and set the Linux link flags. buildsmoke now runs for real in CI.
The standalone-binary build needs Chez's kernel dev files (libkernel.a,
scheme.h) and a C compiler. A distro chezscheme package ships neither, so the
gate failed on CI (apt installs petite+scheme only). Preflight for the csv dir
and cc and skip cleanly when they're missing — same pattern as certify skipping
without Clojure. Where the toolchain exists (dev machines), it runs as before;
the discovered csv dir is pinned via JOLT_CHEZ_CSV so the build uses exactly it.
The list led with parity (numeric tower, persistent collections, future/agent/
pmap, core.async) framed as divergences. Keep the four real ones — no JVM/Java
interop, no BigDecimal, no STM, the irregex engine — plus the coverage caveat,
and state the parity as parity.
Wire the optimization gate to build mode. inline-enabled? (which gates the
inference + flatten-lets + scalar-replace passes in jolt.passes/run-passes) was
hardwired off, so those passes had never run on Chez at all. host-contract now
exposes a settable hc-optimize? flag; `jolt build --opt` flips it on during app
emission.
Kept off for the default release build for now: the passes are sound by design
(RFC 0005/0006) but unexercised on Chez, so release stays on the proven
var-deref codegen until they're validated against the corpus. --opt is the
opt-in fast path. buildsmoke checks both modes produce the same result.
This does not yet deliver direct call binding — the backend has no direct-link
emission path (every :var call still routes through jolt-invoke/var-deref) and
the inline-ir host stash is still a stub. Those are the remaining stage-4 levers.
