jolt build -m ns on a namespace with only top-level side effects and no
-main produced a binary that printed its output, then crashed calling a nil
-main ("nil cannot be cast to IFn"). The launcher now calls -main only when
the entry ns actually defines one; otherwise the top-level forms (already run
at heap build) are the whole program and it exits cleanly. Regression case
added to build-smoke.sh.
A registered #tag data reader whose fn returns a FORM (borkdude/html's #html
expands to (->Html (str …))) was rewritten to a runtime call (reader-fn 'inner),
so the returned code became a runtime list value instead of being compiled —
(str #html [:div]) rendered the code, not "<div>". Clojure applies a data reader
at read time and substitutes its result as code.
loader.ss now applies the reader at load time: a code form (a list) is spliced in
to be compiled, a value (time-literals #time/date -> a Date) keeps the runtime
call, which also keeps a non-serializable constant out of an AOT build. The build
emit path never applied data readers at all (a #tag literal failed a `jolt build`
with "unsupported form"); emit-image.ss gets an ei-emit-form-hook the build sets
to the same rewrite, left as a no-op elsewhere so the seed mint (which doesn't
load loader.ss) is unaffected and the self-host byte-fixpoint holds.
Also make clojure.test report the actual values of a failing (is (= a b)) — it
printed only the form. Restricted to the common pure predicates so a macro head
still takes the plain path.
Fixture test/chez/datareader-app + a smoke check (interpreted) and a build-smoke
check (AOT). make test green, no corpus change.
* Reader records source line/column on list forms
The reader stamps 1-based :line/:column metadata on every list form (plus
:file when load-jolt-file is reading a file), and jolt.host/form-position
reads it back so the analyzer's :pos scaffold finally gets real data. A
left-to-right cursor counts newlines over the delta between successive forms,
so it stays O(n). Vector/map/set literals are untouched (their metadata is a
runtime value the analyzer would have to wrap in with-meta); empty () can't
carry meta. ^meta now merges onto the position keys instead of clobbering them.
Re-mint is byte-identical (the backend doesn't emit :pos), so this is a pure
scaffold for the error-location work that follows.
* Report source location on uncaught errors
Each top-level form records its source position (thread-local) before it
compiles+evals, and cli.ss jolt-report-uncaught appends 'at file:line:col'
when an error propagates out. Covers joltc -e, joltc run <file>, and
load-string — every interpreted path. Top-level granularity, one set per
form; deeper frames come from the Phase 2 frame walk.
Runtime .ss only, no re-mint.
* Clojure stack traces via source registry + native frame walk
A direct-link build emits (jolt-register-source! short-name ns name file line)
once per fn def — at definition time, so zero per-call cost. On an uncaught
error the reporter walks Chez's native continuation frames (jolt-throw captures
the live continuation via call/cc; host conditions carry their own
&continuation), maps each frame's procedure name through the registry, and
prints a Clojure backtrace 'ns/name (file:line)'. Wired into both the cli and a
built binary's launcher.
Frames are keyed by the short munged fn name Chez actually reports (emit-fn's
letrec self-binding), not jv$ns$name; a cross-namespace collision degrades to
the bare frame name rather than a wrong attribution. The analyzer carries the
original form's position through defn macroexpansion onto the def node.
Calling a non-fn now throws a catchable ClassCastException (via jolt-throw)
naming the operator, instead of a raw Chez error.
Caveats (documented in source-registry.ss): names map only in direct-link/AOT
closed-world builds — the open-world -e/repl/run path falls back to the
top-level location; and pervasive TCO erases tail-call frames, so a mapped
trace shows only the non-tail spine. JOLT_DEBUG_FRAMES dumps raw frame names.
Re-mint (analyzer + backend); prelude byte-identical (direct-link off during
mint). Corpus rows certified, build-smoke asserts the trace.
* Propagate source position through macroexpansion
hc-expand-1 now carries the macro call form's :line/:column onto the top of a
list expansion that has none of its own (merged under any meta the macro set),
so errors and stack traces in macro-generated code point at the call site —
Clojure parity. The analyze recursion re-expands inner macros, so each level's
top form picks it up, matching the reference compiler. (meta (macroexpand-1
'(when x y))) now reports the call-site line.
A direct-link fn defined through a user macro (build-app's defguarded) registers
with a real line, so build-smoke's trace assertion covers macro-defined fns.
Runtime .ss (host-contract.ss) — no re-mint; selfhost holds.
Phase 3's optional items are deferred: :line-in-ex-data has no clean consumer
(it would pollute ex-data, break = and printing, and positions already surface
via the trace + top-level location), and Chez source-object emission is a large
backend change the jv$-name registry already sidesteps.
* Review fixes: registration key, thread-locals, debug flag timing
- Register a fn under the name Chez actually reports for its frame, not the def
name: a named fn literal whose name differs from the def (def foo (fn bar …))
is framed as 'bar', and an anonymous fn def (def foo (fn …)) as jv$ns$foo.
Both previously registered under the def name and so never appeared in traces.
- rdr-source-file / rdr-pos-cursor are thread parameters, so concurrent compiles
(futures, core.async) don't clobber each other's file/line attribution.
- Read JOLT_DEBUG_FRAMES at call time: a built binary evaluates top-level forms
at heap-build time, where a load-time getenv is always unset.
Re-mint (backend + reader); prelude byte-identical, selfhost holds.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
A (defmethod m …) where m is :refer'd from another ns passed the bare symbol to
defmethod-setup, which resolved it in the current ns and created a shadow multifn
— the method never reached the real one. Resolve an unqualified name through the
refer table (then current ns) so it lands on the referred multifn.
The AOT build strips the ns form, so the refer table is empty in a binary; emit
chez-register-refer!/-refer-all! per app ns alongside the existing alias
registrations. build-app's fixture gains a defmethod on a referred multifn.
The source loader sets the current ns and registers :as aliases per file. The
build flattened every app namespace into one image with no such markers, so all
app forms ran under the last-set ns ("user"). Two breakages followed, both only
in a built binary:
- defmulti/defmethod resolve their target var through chez-current-ns, so they
registered the multifn under "user" while compiled var-derefs used the baked
ns — the multifn the app saw was uninitialized ("not a fn nil" on dispatch).
- a quoted alias-qualified symbol (a (defmethod ig/foo …) on an aliased multifn)
resolves its ns through chez-resolve-alias, but the stripped (ns …) form left
the alias table empty, so it landed in ns "ig".
bld-ns-prelude now emits (set-chez-ns! ns) plus chez-register-alias! for each
ns's :as aliases before that ns's forms, in both the normal and tree-shake emit
paths. The build-app fixture gains a :default multimethod and an aliased cross-ns
defmethod so buildsmoke covers both across all build modes.
--tree-shake now shakes the clojure.core/stdlib prelude in the same reachability
graph as the app + libraries — only core fns actually reached from -main ship.
dce-blob-records reads prelude.ss with Chez `read`, unwraps each
(guard ... (def-var! "ns" "name" V)) and builds the core->core call graph from the
(var-deref/jolt-var "ns" "name") refs in V — the real emitted edges, no
re-analysis. bld-shake-all merges core + app records into one BFS; roots are -main,
side-effecting forms, and the clojure.core fns the runtime .ss shims reference by
name (enumerated in dce-runtime-core-roots). The shaken core is spliced where the
prelude.ss blob was, in original (tier) order, so load-time deps are preserved.
Bail (reachable runtime resolve) keeps the prelude whole.
Soundness follows Stalin's rule (any reference, value or call, keeps a def live):
dce-collect-refs counts :var and :the-var; non-def registration forms are always
kept (covers protocol/multimethod dispatch). Validated by make shakesmoke: the four
deps.edn git-lib apps build byte-identical output shaken vs not.
Wins (binary): build-app 9.84MB -> 8.12MB (dropped 403/457 core defs); malli-app
10.0MB -> 8.1MB; markdown 9.9 -> 8.3; commonmark 9.8 -> 8.1 — all output-identical.
build-smoke asserts an unused core fn (group-by) is dropped; full make test green.
An AOT-compiled app only needs the analyzer/back end at runtime to compile from
source — eval / load-string / load-file. Macros are expanded at build time and a
require of a baked namespace no-ops, so a closed app that never compiles at runtime
carries the compiler image (~0.8MB) as dead weight.
Under --tree-shake, when reachability is trustworthy (no bail) and no reachable code
references eval/load-string/load-file/load-reader/load, omit host/chez/seed/image.ss
+ compile-eval.ss from the runtime manifest. bld-tree-shake returns the flag
alongside the shaken forms; bld-emit-runtime filters the manifest.
Measured: build-app 9.84MB -> 9.05MB, still runs. Safety verified: an app that evals
keeps the compiler (eval is a bail + compile ref) and eval works at runtime.
build-smoke asserts the compiler is gone in the no-eval app; full make test green.
`jolt build --tree-shake` (or deps.edn :jolt/build {:tree-shake true}) does
reachability DCE over the re-emitted app + library namespaces: keep -main, every
side-effecting (non-def) top-level form, and every def reachable from those; drop
the rest. A macro (expanded at AOT, never called at runtime) is prunable too.
Sound: bails (keeps everything) if REACHABLE code resolves vars by name at runtime
(eval/resolve/ns-resolve/requiring-resolve/find-var/intern/load-string/...), which a
static call graph can't follow. Unreached eval-using library code is simply shaken
away and never triggers the bail. clojure.core and the compiler image stay baked
(prelude + image blobs), so only re-emitted namespaces are shaken for now.
The reachability machinery is in emit-image.ss (records: keep?/fqn/refs/str via
reduce-ir-children) + build.ss (BFS + bail check). build-smoke covers it (drops the
unreachable `twice` macro, output unchanged). Opt-in; default builds are untouched.
full make test green.
Scope note: this shakes the re-emitted app/lib code only. Measurement shows jolt's
compiled code is ~5.8MB of a ~9.8MB binary, dominated by the clojure.core prelude
(~1.5-2MB) and the compiler image (~0.8MB) — both baked blobs this pass doesn't
touch. Those (shake-core, drop-compiler-when-no-eval) are the larger footprint wins,
filed as follow-ups.
Release builds can legitimately want runtime dynamism (redefinition, eval,
load-string), so closed-world direct-linking shouldn't be forced on them. Gate it
behind an explicit --direct-link flag (or deps.edn :jolt/build {:direct-link
true}); off by default in every mode, including release and --opt.
build-binary takes an explicit direct-link? arg instead of deriving it from the
mode. build-smoke now covers the --direct-link path and asserts the cross-ns call
actually lowers to a jv$ binding; default release stays dynamically linked.
A built binary dropped its deps.edn :jolt/native declarations and its
resource roots, so an FFI+resources app (ring-app) failed at runtime:
sockets/sqlite gave 'no entry for socket' and io/resource returned nil.
The buildsmoke fixture is pure compute, so neither path was exercised.
The launcher now loads required + :process native libs before the app's
top-level forms (a library's defcfn resolves its foreign-procedure symbols
at top-level eval during startup, so the libs must be loaded first);
optional libs load in the scheme-start launcher, where a missing lib is
caught rather than aborting the heap build.
deps.edn :jolt/build {:embed [dirs]} bakes those dirs' files into the heap
(register-embedded-resource! at heap build), so io/resource serves them with
no files on disk. Non-embedded resources resolve at runtime against JOLT_PWD,
and io/file reads (e.g. config.edn) stay external.
build-binary now takes the encoded natives, embed dirs, and project paths
from cmd-build; deps/resolve-project surfaces them. Buildsmoke fixture grows
an embedded resource + a :process native to cover both paths.
The standalone-binary build needs Chez's kernel dev files (libkernel.a,
scheme.h) and a C compiler. A distro chezscheme package ships neither, so the
gate failed on CI (apt installs petite+scheme only). Preflight for the csv dir
and cc and skip cleanly when they're missing — same pattern as certify skipping
without Clojure. Where the toolchain exists (dev machines), it runs as before;
the discovered csv dir is pinned via JOLT_CHEZ_CSV so the build uses exactly it.
Wire the optimization gate to build mode. inline-enabled? (which gates the
inference + flatten-lets + scalar-replace passes in jolt.passes/run-passes) was
hardwired off, so those passes had never run on Chez at all. host-contract now
exposes a settable hc-optimize? flag; `jolt build --opt` flips it on during app
emission.
Kept off for the default release build for now: the passes are sound by design
(RFC 0005/0006) but unexercised on Chez, so release stays on the proven
var-deref codegen until they're validated against the corpus. --opt is the
opt-in fast path. buildsmoke checks both modes produce the same result.
This does not yet deliver direct call binding — the backend has no direct-link
emission path (every :var call still routes through jolt-invoke/var-deref) and
the inline-ir host stash is still a stub. Those are the remaining stage-4 levers.
Restores the standalone-binary capability the Janet host had. `bin/joltc build
-m NS -o OUT` AOT-compiles an app into a single self-contained executable — the
whole runtime, clojure.core, stdlib and compiler embedded, no Chez install or
jolt source needed at runtime.
Pipeline (host/chez/build.ss, host primitive jolt.host/build-binary driven by
jolt.main's build command): resolve deps, load the entry namespace recording the
app namespaces in dependency order, re-emit each to Scheme, textually inline the
cli.ss runtime load sequence into one flat source + the app + a launcher, then
compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link against
libkernel.a.
Two non-obvious bits: the compile pass runs in a fresh Chez, not the loaded
runtime (regex.ss shadows top-level `error`, which otherwise bakes a broken
reference into the boot); and the launcher installs scheme-start rather than
running -main at top level, since boot top-level forms execute during heap build
before argv is set, so args only reach -main through scheme-start.
Loader: a require of an in-memory namespace with no source file now no-ops, so
AOT'd app namespaces satisfy require in a built binary.
Mode flags (--opt/--dev, default release) are plumbed; the optimization passes
they gate come in a later stage. RFC 0007 has the design. Gated by `make
buildsmoke`.
