host/chez/build-joltc.ss builds joltc into target/<profile>/joltc: it emits a
flat source of the full runtime + compiler image + inlined build.ss + every
jolt-core/stdlib file as a baked string literal + a cli.ss-style launcher, then
(in a fresh Chez, so the inlined runtime's redefinition of error doesn't strand
early references and runtime eval still sees the runtime's top-level procedures)
compiles it and cc-links it with the Chez petite/scheme boots and the launcher
stub embedded as C arrays. The launcher reads those arrays via FFI on
(jolt-materialize-bundles!) and registers them so build-self-contained can spill
them. joltc itself is cc-linked (clean signature for Homebrew); only the apps it
later builds use the appended-stub path.
build.ss: skip the csv toolchain check on the self-contained path and create the
build dir with a subprocess-free bld-mkdir-p, so a from the
distributed binary shells out to nothing.
release = optimize-level 3 + no inspector info + compressed; debug =
optimize-level 0 + inspector + procedure source + debug-on-exception.
joltc-selfbuild-smoke.sh (make joltcsmoke) builds joltc and, with an empty
environment (no chez/cc/PATH), drives it through the build-app fixture, asserting
the produced binary's output. .gitignore ignores target/.
A record-or-nil (a protocol method whose impls return a record in one branch and
nil in another, or an `if` over a ctor and nil) now types as a NILABLE record
instead of widening to :any. A nilable record still bare-indexes its field reads
(jrec-field-at falls back to jolt-get on nil), but some?/nil? do NOT fold on it, so
a runtime guard is preserved — and inside (if (some? x) ..) / (if x ..) the then-
branch narrows x to the non-nil record, so its reads bare-index AND unbox there.
This is what lets the bounced ray type without a hint: scatter returns
ScatterResult-or-nil (Metal absorbs some rays), and the consumer reads
(:ray scattered) only under (if (some? scattered) ..). The narrowing proves
scattered non-nil there.
lattice: :nil type; :nil ∨ struct -> nilable struct, ∨ anything else -> :any;
nilability is contagious through a struct join, which also now preserves :type when
both sides agree (needed so a record ∨ its nilable self stays that record).
truthy-type?/field-type/pred-on treat a nilable struct as maybe-nil. types: nil
literal -> :nil; an `if` whose test is (some? x)/(nil? x)/x narrows the nilable
local x in the proven branch.
Ray tracer with NO hints: 38.4s -> 23.9s (~1.6x) — hit-sphere now types fully
(0 jolt-get, 57 jrec-field-at, 38 fl-ops), identical to the hand-hinted build.
run-narrow.ss gate, incl. the load-bearing check that the nil case still takes the
else branch (the guard is not folded away). make test / shakesmoke green, selfhost
holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method whose impls all return the same record type has a monomorphic
return. collect-pm-rets! scans the unit's (register-(inline-)method ..) forms,
infers each impl fn's return type, and joins them per method; call-ret-type then
types a (method recv ..) call as that record, so a field read off the result
bare-indexes — e.g. (:ray (scatter m ..)) reads off a Ray. A disagreeing impl
joins to :any and keeps the generic path.
run-protoret.ss: a method with all-record impls bare-indexes + unboxes the field
read; a mixed-return method (one impl returns a number) stays generic. make test /
shakesmoke green, selfhost holds, 0 new divergences.
Foundation for auto-typing record values that flow through protocol dispatch. Does
not yet move the ray tracer: its scatter returns ScatterResult-or-nil (Metal
absorbs some rays), and the nil widens the join to :any — typing a nullable return
soundly needs flow-sensitive narrowing (a guarded (some? x) proves non-nil), filed
separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
A record field tagged ^double now reads back as a flonum and feeds the numeric
pass, so hintless arithmetic over those fields lowers to fl-ops — the leaf-numeric
analog of the ^Vec3 nested-field hints. Combined with the whole-program :double
param inference, a vec3-dot over a ^double-fielded record unboxes end to end with
no per-fn hints.
records.ss: a ^double field tag passes through resolution, and the ctor (and a
mutable-field set!) coerce a ^double field to a flonum — JVM primitive-field
parity (jolt returned an exact 1, not 1.0, before), and what makes reading the
field back as :double sound for an fl-op.
types.clj: field-type-from-tag maps "double" -> :double, and a keyword/get lookup
whose result is :double annotates the node :num-read :double. numeric.clj reads
that annotation and classifies the field read as a :double operand, so the
enclosing arithmetic specializes — the read itself keeps its jrec-field-at/jolt-get
emit.
run-fieldnum.ss gate: ctor coercion (int field -> flonum), field-field arithmetic
emitting fl*/fl+, and an untagged field staying generic. make test / shakesmoke
green, selfhost holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
The closed-world fixpoint (#226) flowed record types across fn boundaries; this
adds a numeric refinement so a hintless fn whose every call site passes a flonum
has its param unboxed to fl-ops, no ^double hint needed.
Lattice gains :double, a flonum refinement of :num: two doubles join to :double,
a double joined with anything else widens to :num — so a param is :double only
when every contributing value is a flonum, which is what makes the fl-op sound.
infer types a flonum literal and flonum arithmetic (+ - * / min max inc dec over
double/int-literal operands) as :double, and the fixpoint joins those across call
sites and return types like any other lattice value.
The bridge to the existing hint-directed pass is a synthetic [param :double]
nhint: wp-infer! stashes the :double params separately from the structural seeds,
and run-passes injects them as nhints before numeric/annotate, so the fl-op
emission and the exact->inexact entry coercion (a no-op on a proven flonum) apply
unchanged.
Sound subset only: :double, never :long — an untyped integer can be a bignum and
fx-ops would overflow/diverge from jolt's arbitrary precision. So an integer
caller leaves a param generic; an escaped fn (unknown callers) keeps :any.
run-numwp.ss gate: cross-fn :double propagation incl. through a flonum-returning
helper, the integer-caller and escape negatives, and the full run-passes path
emitting fl* + entry coercion. make test / shakesmoke green, selfhost holds, 0
new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the inference types a keyword-lookup receiver as a record — it carries the
field-order :shape and :hint :struct from the whole-program fixpoint — the back
end reads the field by its declared slot via jrec-field-at instead of jolt-get.
That skips the jolt-get case-lambda, the dispatch fn, and the field-key
hashtable lookup, leaving a jrec? check + a static-index vector-ref.
jrec-field-at falls back to jolt-get when the receiver isn't the expected record
(a map downgraded by dissoc, or a value the inference mistyped), so it stays
correct if the static type is wrong. Only the no-default form takes the bare
path (a declared field is always present).
Sound only for non-nil records: a self-recursive param that can be nil (e.g.
binary-trees check-tree, whose untagged child is nil at leaves) types :any and
keeps jolt-get — the whole-program fixpoint demotes it. The target is non-nil
record params, like a Vec3 dot product (~5% there; boxed-flonum arithmetic
dominates the rest, a separate numeric lever).
run-fieldread.ss gate: emitted form uses jrec-field-at at the right slot and
matches jolt-get for each declared field; a non-field key and a default-arg form
keep the generic path. make test / shakesmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the inference proves a protocol call's receiver is one record type, the
back end resolves the impl by that static tag (find-protocol-method) instead of
routing through the protocol var -> jolt-invoke -> protocol-resolve, which
re-derives the tag and walks the type table. Same table lookup, minus the
var-deref, the rest-cons, and the receiver-type computation.
Fires only on a monomorphic site: a megamorphic receiver joins to :any and
carries no :devirt-type, so it keeps ordinary dispatch (the dispatch bench is
unaffected). The annotation comes from the whole-program fixpoint typing a
reduce/HOF element or a ctor return as a specific record.
Modest on the dispatch benchmarks (~6% on mono-dispatch) — float boxing in the
reduce accumulator dominates there, a separate numeric lever — but it removes
the dispatch overhead wherever a typed receiver is known.
run-devirt.ss gate: emitted form uses find-protocol-method, and evaluating it
matches ordinary dispatch for an inline impl, an extend-type impl, and the
non-devirt path. make test / shakesmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
Re-derive each app fn's param types from its call sites under --opt, so a
record type flows across fn boundaries: a ctor's return reaches a callee
param, and a typed vector's element reaches a HOF closure's param. The back
end can then bare-index field reads and devirtualize protocol calls at those
sites (it reads the resulting :hint/:devirt annotations; consuming them is
separate work).
This rebuilds the inter-procedural driver the Janet host had — the API
(infer-body/reinfer-def) survived the rehost but nothing drove it, and the
record-shapes/protocol-methods registries were empty stubs.
- records.ss: populate record-shapes (ctor key -> fields/tags/type, resolving
nested record field tags) and protocol-methods (method var -> [proto method])
registries at deftype/defprotocol load time; jolt.host accessors materialize
them.
- passes/types.clj: wp-infer! runs a closed-world fixpoint joining call-site
arg types into callee params; reinfer-def re-seeds each def at emit. Self-
recursive calls and fn-level recur are collected so a recursive fn's params
are constrained by its recursion, not just external callers — else a param
the recursion widens (e.g. binary-trees check-tree, whose untagged child can
be nil) would be unsoundly typed non-nil. A fn used in value position keeps
:any params (callers unknown). Megamorphic sites join to :any.
- build.ss: analyze all app forms and run the fixpoint before per-form emit.
- run-wp.ss: gate (cross-fn propagation, escape soundness, self-recursion).
make test / shakesmoke green, 0 new divergences, selfhost holds.
Co-authored-by: Yogthos <yogthos@gmail.com>
Two things, both from studying Stalin's closed-world DCE:
1. Soundness fix: dce-collect-refs now counts a :the-var (#'x / (var x)) reference,
not just a :var. Stalin's rule is that ANY reference — value position, not only a
direct call — keeps the target live; a var referenced as #'x would otherwise be
wrongly dropped. (My :var collection already covered value-position refs; this
closes the the-var hole.)
2. host/chez/tree-shake-smoke.sh (make shakesmoke): builds example apps default vs
--tree-shake and requires identical output — the real risk is shaking a binary
that pulled libraries via deps.edn. Covers markdown/malli/commonmark/hiccup
(git-lib apps). All produce byte-identical output shaken vs not, and drop
~0.8-1MB (malli 10.0MB -> 9.0MB) from the compiler-drop. Slow; not in the default
gate. Skips without the examples repo.
jolt.passes.inline was fully written but dormant — it fetched bodies via the
inline-ir host hook, which was a stub returning nil. Wire it up: run-passes stashes
each inline-eligible defn (single fixed arity) as its form is optimized, and
inline-ir hands the body back at call sites under --opt.
The catch was the ^double/^long coercion: an inlined fn drops its param-entry and
return coercion, so (work 3 4) on a ^double fn would return 25 instead of 25.0. New
:coerce IR node carries the coercion inside the spliced body — the inline pass wraps
a hinted param's arg and the return in :coerce, the back end lowers it
(exact->inexact / jolt->fx), and jolt.passes.numeric reads its :kind. So an inlined
call matches the called one and the body's fl*/fx* fast path still fires.
Only under --opt (closed world); the seed mint and -e don't inline, so selfhost and
the corpus are unaffected. test/chez/inline-test.ss 12/12 (make inline); full make
test green, 0 new corpus divergences.
Bench (hot loop, body is a ^double helper call): direct-link 500ms -> --opt
(inlined) 184ms = 2.7x, by eliminating the call + coercion wrappers and letting Chez
fuse the fl-ops unboxed. ~26x over the default dispatched build.
A ^double/^long param hint (or a float literal) now drives Chez flonum/fixnum
ops instead of generic arithmetic — JVM-style primitive hints, available in every
build and at -e (not gated on direct-linking or whole-program inference).
New pass jolt.passes.numeric: a local forward type-flow seeded from ^double/^long
fn-param hints (analyzer attaches :nhints per arity) and float literals,
propagated through let inits / arithmetic / if / do. It tags an arithmetic invoke
:num-kind :double|:long when every operand is that kind (an integer literal is a
wildcard, coerced to a flonum in a double op). The back end lowers a tagged node
to fl+/fl-/fl*/fl//fl<?/... or fx+/fx*/fx1+/fxquotient/... (unchecked-add etc.
join the fixnum path; == too). Runs last in run-passes, both branches.
Soundness: :long is seeded ONLY from an explicit ^long hint, never a bare integer
literal, so un-hinted integer code keeps jolt's arbitrary-precision numbers — no
fixnum-overflow surprise, no corpus divergence. :double comes from ^double hints
and float literals (flonum arithmetic is always flonum, matching the generic
result). A ^long hint is a promise the value is a fixnum: fx+ raises on overflow,
like a JVM fixed-width long.
Numeric-hinted params coerce at fn entry (exact->inexact / jolt->fx), the way the
JVM coerces a primitive parameter — so the body's fl*/fx* ops can rely on the
type even when a caller passes an exact int (e.g. Chez's (* 0 1.0) => exact 0).
Round 1 specializes hinted straight-line / fn-body arithmetic. fl-ops are ~4x
generic in a tight Chez loop, but realizing that on loop-carried accumulators
needs loop-var typing — round 2. Sound foundation, gated by test/chez/numeric-test.ss.
A release/optimized `jolt build` is a closed world: every app def is final, so
an app->app call can bind to the def's Scheme binding directly instead of going
through (jolt-invoke (var-deref ns name)).
The emitter gains a direct-link mode (off for the seed mint, runtime -e/repl, and
dev builds). With it on, a top-level app def also emits a binding jv$<ns>$<name>
that def-var! aliases; an app->app call or value-ref to a name already emitted in
the unit lowers to that binding, skipping both the var-table lookup and the
generic IFn dispatch. ^:dynamic/^:redef defs and nested defs (a defonce's inner
def) opt out and stay indirect. Off direct-link mode, emit-top-form is exactly
emit, so the seed and runtime eval are byte-unchanged (selfhost holds).
build.ss turns it on for release + optimized; the defined-set accumulates across
the dependency-ordered namespaces so a dep's defs are linkable by the time the
entry that calls them is emitted. App->core calls stay indirect for now (core is
the baked seed); that's a later stage.
~1.74x on a hot cross-namespace call loop (26.5s -> 15.2s).
The corpus/unit gates compile through run-passes' const-fold-only branch, so
the type-inference walk runs only under jolt build --opt — buildsmoke hit one
trivial app and checked stdout. run-infer.ss drives the pass directly: analyze
a source string, then call check-form / infer-body / the set-*! registries and
assert diagnostic counts and collected calls/escapes. Wired into make ci.
Gives the inference pass real behavior coverage so refactoring its internal
state is gate-validatable. jolt-ogib.10 groundwork.
Restores the standalone-binary capability the Janet host had. `bin/joltc build
-m NS -o OUT` AOT-compiles an app into a single self-contained executable — the
whole runtime, clojure.core, stdlib and compiler embedded, no Chez install or
jolt source needed at runtime.
Pipeline (host/chez/build.ss, host primitive jolt.host/build-binary driven by
jolt.main's build command): resolve deps, load the entry namespace recording the
app namespaces in dependency order, re-emit each to Scheme, textually inline the
cli.ss runtime load sequence into one flat source + the app + a launcher, then
compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link against
libkernel.a.
Two non-obvious bits: the compile pass runs in a fresh Chez, not the loaded
runtime (regex.ss shadows top-level `error`, which otherwise bakes a broken
reference into the boot); and the launcher installs scheme-start rather than
running -main at top level, since boot top-level forms execute during heap build
before argv is set, so args only reach -main through scheme-start.
Loader: a require of an in-memory namespace with no source file now no-ops, so
AOT'd app namespaces satisfy require in a built binary.
Mode flags (--opt/--dev, default release) are plumbed; the optimization passes
they gate come in a later stage. RFC 0007 has the design. Gated by `make
buildsmoke`.
The HTTP server moves out of the host into the jolt-lang/ring-janet-adapter
library, which binds sockets itself via jolt.ffi and shuts down cleanly. Drop
host/chez/http-server.ss and the obsolete ffi-server-test (the FFI collect-safe
path is covered by ffi-binding-test; the server by the adapter's own CI).
A jolt library can now bind its own native dependencies and expose a Clojure API
over them — no jolt built-in required. This is the foundation for moving the
http-client / db / adapter functionality out of the host and into real libraries.
- jolt.ffi/foreign-fn (sugar: defcfn) is a compiler special form: a compile-time
-typed C signature lowers to a real Chez foreign-procedure (analyzer :ffi-fn ->
backend foreign-procedure), so calls are typed and marshaled, not eval'd.
- host/chez/ffi.ss provides the rest under jolt.ffi: load-library, alloc/free,
read/write/sizeof, ptr<->string, null/null?. Loaded after the loader snapshot
so a library's (require '[jolt.ffi]) still loads the macro side.
- Types: int/uint/long/ulong/int64/uint64/size_t/ssize_t/iptr/uptr/double/float/
pointer/string/void/uint8/char.
Validated end to end: a pure-Clojure file binds libc (getpid/strlen/abs) and
libsqlite3 (open/prepare/step/column/finalize over out-param pointers) and runs a
query. Gate test test/chez/ffi-binding-test.ss (make ffi); selfhost holds.
The Chez port had landed transients as copy-on-write — each conj!/assoc!/etc.
rebuilt the whole persistent collection. Semantics were right but a transient
vector was O(n^2) to build (the persistent vector is a flat array, so every
conj! copied it); maps/sets were ~O(n log n) since the HAMT only path-copies.
This restores the Janet host's approach: true mutable backing, snapshot once on
persistent!.
vec : a growable Scheme vector (capacity + fill count); conj!/pop! amortized
O(1), persistent! hands off the buffer (exact fit) or trims once.
map : a Chez hashtable keyed by key-hash/jolt= (value equality, nil-safe);
persistent! folds it into a pmap.
set : a Chez hashtable; persistent! folds into a pset.
cow : fallback for anything else (e.g. a sorted coll) keeps the old
copy-on-write path, preserving jolt's superset.
get/count/contains?/nth see through each representation. Building a 400k vector
went from minutes (quadratic) to ~50ms (linear). assoc! keeps the variadic
dangling-key nil-pad on both vectors and maps. test/chez/transient-test.ss pins
the invariants and the linear-time property; wired in as `make transient`.
Two thread-safety bugs in the native FFI layer.
The HTTP server's accept/recv/send were plain foreign-procedures. A thread
inside a foreign call stays active for the stop-the-world collector, so the
accept loop sitting idle in accept() froze GC for the whole process whenever
another thread (a future, an async block) allocated. Mark the three blocking
calls __collect_safe so the thread deactivates for the call's duration —
collection proceeds while the accept thread waits. The args are an fd and
foreign-alloc'd buffers (outside the Scheme heap), so a collection mid-call has
nothing to move.
jolt.http-client built its -D header-file path from an unguarded (set! counter
(+ counter 1)) and counter mod 90000, with no per-process component. Concurrent
requests could compute the same path and clobber each other's headers. Use a
mutex-guarded monotonic counter plus the pid.
test/chez/ffi-server-test.ss exercises both (a (collect) while the server is
idle in accept(), temp-path uniqueness across threads, and a live request) and
is wired into the gate as `make ffi`.
The self-host byte-fixpoint (make selfhost) only holds on the Chez that minted
the seed — CI's Debian Chez emits byte-different output for some constructs
(isolated to the dedupe re-mint), so it failed there. The checked-in seed RUNS
correctly on any Chez, so CI now runs 'make ci' (corpus/unit/smoke/sci/certify);
'make test' keeps selfhost for local dev. Cross-version emit determinism tracked
in jolt-8479.
Re-port the SCI compatibility stress test to joltc: host/chez/run-sci.ss loads
borkdude/sci's own source (vendor/sci, re-vendored) through the spine and
requires its forms to compile+eval. Floor-gated at 160/218 forms (the tail is
genuine host gaps — set! on vars, some macro/def shapes); raise as they close.
Wired into 'make test' (skips if the submodule isn't checked out).
jolt-cf1q.6
Remove the Janet seed (src/jolt/*.janet: reader, value layer, vars/ns, the
tree-walking interpreter, the Janet backend, the optimizing compiler), the
Janet->Scheme cross-compiler (host/chez/{driver,emit,jolt-chez}.janet),
bin/jolt-chez, the jpm build (project.janet) and the Janet test runner
(run-tests.janet), plus the entire Janet test suite. jolt now builds and runs
on Chez alone: bin/joltc off the checked-in seed, bootstrap.ss to rebuild it.
The portable Clojure stays: jolt-core/**, host/chez/**.ss, and the stdlib +
tooling under src/jolt/clojure + src/jolt/jolt (read by the seed build, no
Janet). The gate is 'make test' (self-host, corpus, unit, cli smoke, certify).
Drop the sci and clojure-test-suite submodules (used only by deleted Janet
integration tests); irregex stays.
Filesystem corpus/unit cases that probed project.janet now probe README.md.
jolt-cf1q.6
Add a Janet-free gate so correctness can be judged with only Chez + Clojure:
- host/chez/run-corpus.ss: corpus.edn vs JVM expecteds, lifting the per-case ns
isolation from the old Janet driver; reads corpus.edn via the jolt reader.
- host/chez/run-unit.ss + test/chez/unit.edn: the host-specific unit cases,
evaluated in-process and compared to baked expecteds.
- host/chez/selfcheck.sh: self-host fixpoint (bootstrap.ss rebuild == checked-in seed).
- host/chez/smoke.sh: real bin/joltc CLI smoke.
- host/chez/remint.sh: re-mint the seed to a byte-fixpoint after a source change.
- Makefile: 'make test' runs the lot; 'make remint' rebuilds the seed.
Numbers match the Janet gate: corpus 2679/2757 0 new div, unit 450/450, certify
0 new/0 stale.
jolt-cf1q.6
