Ports the full # dispatch to the portable reader: #{} sets, #() anon-fns, #?/#?@
reader-conditionals, #_ discard, #' var-quote, #"" regex, #inst/#uuid/#tag tagged
literals, ## symbolic (Inf/-Inf/NaN), and #^ deprecated metadata. With this the
reader is feature-complete except position tracking + wire-in (inc 5d).
Reader-conditionals resolve clause-order against a portable feature set (atom
#{:jolt :default}); #? -> :skip / :form, #?@ -> :splice (the control protocol from
5b). #() uses the two-pass %-scan (collect indices, then rebuild replacing %N/%/%&
with gensym params) over the form tree via the jolt.host form-* contract. Three
host constructors added: form-make-set, form-make-tagged, form-gensym-name.
reader-parity 149/149. #() compares modulo gensym (canonicalize #-suffixed param
names by first-occurrence order — the two readers gensym different names but the
structure + %-mapping must match). ##NaN checked by the NaN!=NaN property. Full jpm
gate green (prelude pre-warmed). jolt-9ufe.
Ports list/vector/map literals and the quote family (' ` ~ ~@ @) + metadata (^)
to the portable Clojure reader. read-form now returns a [kind payload pos] control
triple (:form / :skip / :splice) instead of the Janet reader's :jolt/skip sentinel
FORMS — out-of-band control is collision-free and host-neutral (no tagged struct
to build or recognize). read-delimited dispatches the kinds; read-next-form skips
comments where a single datum is needed; read-map pairs k/v skipping trivia in
either slot. syntax-quote of a self-evaluating literal collapses at read time.
Four host constructors added to the contract (host_iface): form-make-list/vector/
map + form-sym-merge-meta (attach ^meta to a symbol). form-make-map reuses the
seed's reader-map (now public) for the source-order kv tracking. The portable
reader accumulates items in a jolt vector and the host builds its native form rep.
Gate: reader-parity 107/107 (lists/vectors/maps incl. nested + comments-in-coll,
quote/syntax-quote-collapse/unquote/deref, ^:dynamic/^Type/^{} meta). Full jpm gate
green (prelude cache pre-warmed — a cold cache races under the parallel gate when
the jolt-chez fingerprint changes; pre-existing, see new bead). jolt-sh1n.
fix-bugs-dont-reproduce, scoped per the keeper rule: jolt-if19 (a leading + on a
numeric literal errored instead of reading as the positive number) is fixed in
jolt.reader (read-number* now strips a leading + like -, positive), the code we
keep. The Janet seed reader (reader.janet) is left untouched — it's deleted in
Phase 5, so fixing it is wasted work.
Since the seed reader stays buggy, reader-parity can't use it as the oracle for
these inputs: added check-correct to assert the portable reader against the hand-
verified value (+5 => 5, +42, +0xff => 255, +3.5). reader-parity 67/67. No Janet
binary/gate impact (jolt.reader is not yet in the binary path). jolt-if19.
Starts taking the reader off Janet (src/jolt/reader.janet, 831 lines) into
portable jolt-core Clojure. jolt.reader holds the lexing/parsing LOGIC; form
construction + string->number parsing delegate to the jolt.host contract — a
Clojure source file can't write a {:jolt/type :symbol} literal (parses as a
tagged form) and the concrete representation is the host's to own. Same split the
analyzer/emitter already use. Once cross-compiled this runs on Chez so compile-
from-source needs no Janet reader.
inc 5a = the atom layer: whitespace/comments, symbols (+ nil/true/false),
keywords, strings (escapes), numbers (sign/hex/radix/ratio/fractional/exponent,
trailing N/M), characters. Collections, quote/deref/meta and dispatch (#) follow
in 5b/5c (throw not-yet-ported). Positions are char indices (Janet uses bytes);
identical for ASCII and the gate compares form VALUES, not positions.
host_iface.janet gains four reader primitives on the contract: form-make-symbol,
form-make-char, form-char-from-name, form-scan-number (the irreducible host bits
the portable reader rests on). Additive — new jolt.host interns, nothing else
changed.
Surfaced jolt-if19 (Janet seed reader: +N literals error instead of reading as N;
read-number strips only the - sign). The port reproduces it; both-throw counts as
faithful parity in the gate.
Gate: reader-parity 64/64 (symbols/keywords/strings/ints/hex/radix/ratio/floats/
exponent/N-M/chars). Full jpm gate green after clean rebuild, conformance 355x3.
jolt-50xx.
driver.janet now compiles IR via the portable Clojure emitter (jolt.backend-
scheme) instead of emit.janet, at every entry point (compile-program, emit-core-
prelude, eval-e-with-prelude). The emitter is loaded into the ctx and called like
the analyzer. emit.janet stays only as the emit/program string-wrapper until
program assembly ports to Clojure with compile-from-source; its emit fn is no
longer called anywhere (emit-test's truthy-elision helper now uses the new
d/scheme-emit too). This takes the IR->Scheme emitter off Janet.
A form-by-form diff of the two emitters over the whole prelude found one gap:
emit-const missed char literals because a :jolt/type-tagged struct is not a plain
jolt map? — switched to the form-char? host contract. Diff then 0.
jolt-chez prelude fingerprint now includes backend_scheme.clj + host_iface.janet.
Gate: full prelude corpus 2280/2494, NEW divergence 0, same buckets as the Phase-2
emit.janet floor (36 emit-fail, 170 crash) — the Clojure emitter is byte-for-
behavior identical. emit-test 331/331 (now via the Clojure emitter), emit-parity
58/58. jolt-duot.
Completes the op coverage of the portable Clojure emitter — it now handles every
op emit.janet does (const/local/var/the-var/host/host-static/host-new/if/do/
invoke/vector/set/map/quote/throw/try/regex/inst/uuid/host-call/let/loop/recur/
fn/def). Adds emit-try (guard + dynamic-wind), :throw, :regex/:inst/:uuid, and
:host-call (jolt-host-call for rt-shimmed methods else record-method-dispatch).
def-meta + quoted-symbol-meta needed emit-quoted to reconstruct plain jolt VALUES
(metadata maps), not just reader forms. The blocker was that :meta arrived as a
raw Janet table embedded in the IR — jolt's count/map?/keys don't work on a table
(counter to jolt.ir's 'no host values embedded'). Fixed at the host seam:
h-sym-meta now returns the meta as an immutable struct, which is a portable jolt
map (jolt count/map?/keys work on a struct, and the Janet backend's merge/get
still do too). emit-quoted handles both reader forms (jolt.host form-* contract)
and jolt-value collections (native map?/vector?/set?/seq? branches, after the
form-* branches so reader forms win).
Gate: emit-parity 55/55 (incl try/catch/finally, ^:private def-var-with-meta!
structural check, inst/uuid eq, regex smoke, quoted-sym-meta). Full jpm gate
green after clean rebuild (seed change). jolt-me6m.
Adds :vector/:map/:set (emit-ordered, left-to-right element eval) and :quote to
the portable Clojure emitter. Collection-literal nodes carry already-analyzed IR
items so they just recurse; quote walks the RAW reader form.
emit-quoted walks the reader form via the jolt.host form-* contract (form-list?/
form-elements/form-vec-items/form-map-pairs/form-set-items/form-sym-*), the same
portable seam the analyzer uses — not host-native predicates, so it works
unchanged whether the form came from the Janet reader or the Chez reader. Reader
forms are raw host representations (Janet list=array, vec=tuple), so native
list?/vector? would not see them; the contract is the correct abstraction and
keeps the emitter host-neutral. Quoted-symbol metadata and def-meta still defer
to inc 3.
Surfaced a latent Janet-host bug (jolt-tg9s): (quote #{...}) evaluates to the raw
reader form instead of a reconstructed set, so (contains? (quote #{:p :q}) :p) is
false on build/jolt. The Chez emitter is correct (real Clojure: true); the parity
test asserts the verified value for those two cases.
Gate: emit-parity 42/42 (incl vector/map/set literals, coll/kw-as-fn, quoted
list/vec/map/set/symbol/nested). emit-test 331/331, conformance 355x3. jolt-7jvp.
First spine increment of self-hosting the compiler on Chez. The IR->Scheme
emitter is host/chez/emit.janet (Janet); to get the analyzer emitting its own
code on Chez with no Janet, the emitter logic has to be portable Clojure that
cross-compiles and runs on Chez itself.
jolt-core/jolt/backend-scheme.clj ports the core ops: const/local/var/the-var/
if/do/let/loop/recur/invoke (+ native-ops)/fn/def, plus the chez-str-lit/flonum/
munge/truthy-elision helpers and prelude-mode. Output is Scheme source text, op-
for-op with emit.janet. recur-target/known-procs are dynamic vars (auto-restore,
no throw-leak). Quote, collection literals, try/throw, host interop, regex/inst/
uuid and program assembly come in later increments (they throw not-yet-ported).
Gate: test/chez/emit-parity.janet loads the Clojure emitter interpreted on the
Janet host and runs each case through it -> Chez -> compares to the Janet CLI
oracle. 18/18 incl fib, factorial loop, multi-arity, variadic, higher-order,
#() shorthand, the mandelbrot kernel. emit-test 331/331 (emit.janet path
untouched), conformance 355x3. jolt-hg7z.
The analyzer lowers a #inst/#uuid tagged form to a :inst/:uuid IR leaf, mirroring
the existing :regex node: the Janet back end punts to the interpreter (its
data-readers parse the literal, so seed behavior is unchanged), the Chez back end
emits jolt-inst-from-string / jolt-uuid-from-string.
host/chez/inst-time.ss is the Chez-native value layer: a jinst record holding
epoch ms (RFC3339 parsed via Hinnant civil/days math, with Clojure's partial
defaults and +/-hh:mm offsets), wired into jolt-get (so the overlay inst?/inst-ms
read it), jolt= / jolt-hash (instant identity as a map key), pr-str (#inst
"...-00:00"), str, type, and instance? java.util.Date. The java.time surface
(DateTimeFormatter ofPattern/ISO_LOCAL_DATE_TIME/ofLocalized*, the pattern engine,
Instant, ZoneId, LocalDateTime, FormatStyle, Locale, Date) ports java_base.janet
over host-static.ss's registries.
Corpus 2202->2238, 0 new divergences; clears the whole 'unsupported form'
emit-fail bucket. Full Janet gate green (analyzer/backend changes are
behaviour-preserving — #inst still parses through the interpreter's data-readers
on the seed).
The analyzer lowers the `.` special form (. target member arg*) and the
.-field field-access head to a :host-call instead of leaving them
uncompilable. Janet behaviour is unchanged — its back end punts :host-call
to the interpreter, which re-runs the original `.` form via eval-dot.
The Chez back end routes a non-shimmed :host-call through
record-method-dispatch, extended by a new host/chez/dot-forms.ss with the
arms dispatch-member covers but the record/string base did not, mirroring
src/jolt/interop/collections.janet precedence:
- collection interop first (count/seq/nth/get/valAt/containsKey on a
vector/map/set), so (. {:count 9} count) is the entry count like the seed
- field access for a "-name" member (records and maps)
- the seed's universal object-methods (getMessage/getCause/toString/
hashCode/equals) on a non-record map, winning over a field lookup
- non-record map member: a stored fn is a method called with self, else
the field value
Raw seqs are excluded from coll interop — the seed's behaviour there is
representation-dependent (plain (seq v) vs a lazy-seq) and a normalized cseq
can't mirror it. Also added getMessage/getLocalizedMessage/equals to the
string method surface so a thrown string / Exception. ctor (which keeps the
message string) answers .getMessage.
Parity 2134 -> 2150, 0 new divergences. New test/chez/_dotform.janet 26/26;
emit-test 331/331.
Lower host class interop on the Chez back end. The analyzer now turns a
non-var qualified ref `Class/member` into a :host-static node and a
`(Class. ...)` / `(new Class ...)` form into a :host-new node (ir.clj
gains both, with walker support). The Janet back end punts both to the
interpreter, so its behavior is unchanged (verified: dot-form, `..`
threading, shadowed `new`, and all interop still resolve via fallback).
The Chez emit lowers a value ref to host-static-ref, a call head to
host-static-call, and a constructor to host-new. host/chez/host-static.ss
is the runtime registry these resolve against — the Chez port of the
seed's class-statics / class-ctors / tagged-methods (java_base.janet +
host_io.janet), restricted to the java.lang/util/net/io surface portable
cljc code calls: Math, System (getenv/getProperty/exit/currentTimeMillis),
Long, Integer, Boolean, Character, String, Thread, Class, Pattern
(compile/quote/MULTILINE), URLEncoder/Decoder, Base64, the Number method
surface (byteValue/intValue/...), plus the StringBuilder, StringWriter,
StringReader, PushbackReader, HashMap, StringTokenizer, BigInteger,
String, MapEntry, and exception constructors. Constructed objects are
jhost records dispatched through record-method-dispatch.
Also: emit now evaluates collection-literal elements left-to-right
(emit-ordered) — Chez evaluates call args right-to-left, which had been
swapping side-effecting elements in [(read r) (read r)] and map literals.
This un-allowlisted the 6 eval-order corpus cases (the read-line trio +
the three map-construction cases). Removed `.write` from the
jolt-host-call fast-path so a StringWriter routes through dispatch.
java.time formatting, edn/read-over-readers, and slurp/with-open over
readers are deferred to a follow-up.
Corpus parity 2078 -> 2134 (floor raised), 0 new divergences; the
print-method builtin-override case is allowlisted (same multimethod gap,
newly reachable now that StringWriter constructs). emit-test 326/326,
_javastatic 51/51, conformance 355x3, full jpm test green.
host/chez/multimethods.ss implements the multimethod runtime: defmulti/defmethod
expand to defmulti-setup/defmethod-setup calls (+ get-method/methods/
remove-method/prefer-method/prefers). A jolt-multifn record carries its dispatch
fn and a jolt=-keyed method table; jolt-invoke dispatches it (exact match, then
isa?/hierarchy with prefer-method, then :default), reusing the overlay's
isa?/derive/make-hierarchy. The multifn's ns comes from a runtime chez-current-ns
(default user; the prelude load sets clojure.core for print-method/print-dup).
Two emit-side changes were needed:
- late-bind (:late-bind-unresolved? ctx flag, default OFF): defmulti expands to a
bare-symbol setup call, so the analyzer doesn't intern the name and a forward
reference '(area ...)' after '(defmulti area ...)' in one form was 'Unable to
resolve symbol'. The strict compiler punts these to the interpreter; the Chez
back end has none, so the flag lowers an unresolved symbol to a var-ref against
the compile ns (open-world -e semantics). Set only by the Chez make-ctx /
jolt-chez; the main compiler keeps strict resolution (host_iface late-bind?
defaults nil).
- a :var call head now routes through jolt-invoke, since a late-bound var can hold
a multifn (or keyword/coll IFn), not just a procedure. Transparent for
procedures; the hot self-recursive call is a :local known-proc, stays direct.
Class-based dispatch ((class x)/String) deferred (needs deftype/class subsystem).
Parity 1506 -> 1530/2497, 0 new divergences. emit-test 302/302. Full janet gate
green (the analyzer flag is off there; suite flakiness under parallel load only).
Closes the last clojure.core prelude emit gap (parse-uuid): the whole
non-macro core now lowers to Scheme (prelude reach 355/355).
A #"..." literal analyzes to a :regex IR node. The Chez back end emits
a jolt-regex value over irregex (Alex Shinn, BSD), vendored as the
vendor/irregex submodule -- a portable Scheme regex with PCRE/Java-style
string patterns and first-class Chez support. host/chez/regex.ss wraps
jolt's re-* surface over it: irregex-match -> re-matches (anchored),
irregex-search -> re-find, groups as Clojure [whole g1 ...] vectors,
re-seq as a jolt seq. re-pattern/re-matches/re-find/re-seq/regex? are
def-var!'d into clojure.core so prelude / -e code resolves them.
They stay OUT of the subset native-ops on purpose: irregex's
Unicode/property-class semantics differ from the seed's byte-PEG
approximation, so keeping them prelude-only avoids dragging
engine-difference divergences into the subset-parity corpus. The Janet
back end punts :regex to the interpreter (the seed compiles #"..." to a
Janet PEG), so the main language is unchanged.
Only two adaptations for Chez's top level: a cond-expand shim (Chez's is
library-only) and a normalizing error wrapper (silences irregex's 1-arg
error warnings). rt.ss load is ~0.18s.
emit-test 131/131 (regex literal + re-* parity vs the CLI oracle);
prelude reach 355/355; Chez subset 672/672, 0 divergences; full gate
green.
(.method target arg*) now analyzes to a :host-call IR node instead of
punting at analyze. The Chez back end lowers it to a jolt-host-call
dispatch for the methods the RT shims (.write -> port display,
.isDirectory -> file-directory?, .listFiles -> directory-list); any
other method stays out of subset (clean emit-time reject, so it can't
read as a compiled-but-broken corpus divergence). The Janet back end
punts ALL :host-call to the interpreter, same shape as letfn: compiles
on Chez, interprets on Janet, zero change to the main language.
Closes the io tier's print-method defmethods and file-seq: prelude emit
reach 348 -> 354/355 (50-io 20/20). The one remaining gap is the regex
literal in parse-uuid (needs a regex engine on Chez; deferred).
emit-test 122/122; Chez subset 672/672, 0 divergences; full gate green.
Closes the last two non-host-interop prelude emit gaps.
letfn now analyzes to a :let node flagged :letrec — the binding fns are bound
into the env together before any spec is analyzed, so siblings and self resolve.
The Chez back end lowers it to letrec*; the Janet back end punts it at emit
(its sequential let* can't express the mutual recursion — same interpreter
fallback as before, just decided at emit-ir instead of analyze).
(def x) with no init (declare) analyzes to a :def with :no-init instead of
punting. Chez reserves the var cell via declare-var! (which doesn't clobber an
existing root — (do (def x 7) (def x) x) => 7); the Janet back end still punts
to the interpreter, which interns a genuinely-unbound var.
fallback-zero-test now checks emit-ir too, not just analyze-form, so the real
compile-vs-interpret decision is what it asserts (letfn/def-no-init analyze but
the Janet back end punts them). letfn stays in must-punt with an updated note.
Prelude emit reach 342 -> 348/355 (40-lazy now 13/13); Chez subset 664 -> 672,
0 divergences; emit-test 110 -> 117. Full gate green.
jolt's catch is (catch class binding body*); the binding (3rd element) must be
a symbol. Neither the analyzer nor the interpreter validated it, so a non-symbol
binding crashed with an internal Janet error (expected integer key for array...)
and, in the interpreter, a malformed clause whose body never threw was silently
swallowed (returned the try value). Clojure rejects a non-class/non-symbol catch
clause; match that with an up-front error in analyze-try and eval-try.
Surfaced building the Chez try/throw emit. Regression rows in exceptions-spec
(runs x3 modes) plus a unit test asserting the clean message in interpret and
compile. jolt-kg6p.
The map build already used a transient map, but each bucket was rebuilt with
a persistent (conj (get ret k []) x) per element — an O(log n) trie path
rebuild + alloc each. A coarse grouping (few large buckets) was bound on that
conj, not the map build. Buckets are now native arrays (transient vectors,
O(1) push) frozen once; distinct keys are tracked in a side vector so the
buckets freeze in place with no second map rebuild. A bucket's first element
stays a cheap persistent [x] and only promotes to a transient on the second,
so an all-singletons grouping pays no transient alloc.
coarse (10/100 buckets, 50k): ~313ms -> ~125ms (~2.5x)
2 buckets (50k): ~322ms -> ~129ms (~2.5x)
all-unique (50k): ~949ms -> ~892ms (no regression)
Surfaced a latent bug: canon-key returns nil for a nil key and Janet tables
drop a nil key, so the canon-keyed transient map silently lost a nil-key
entry — group-by/frequencies/assoc!/into{} dropped the whole nil bucket
((group-by identity [nil nil 1]) gave {1 [1]}, not {nil [nil nil], 1 [1]}).
Route nil through a sentinel (tbl-key) at the transient-map keying sites;
persistent!/count/dissoc! work unchanged since the real [nil v] pair is kept
as the stored value, and phm already has its own has-nil slot. The transient
set has the analogous bug (needs phs nil support) — filed separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
into {}, frequencies, group-by, set, into #{} and persistent! all built
their result by folding an immutable assoc/conj per element — each call
rebuilt the O(log32 n) trie path and allocated a fresh wrapper. Add a
one-pass bottom-up HAMT builder (phm-from-pairs) and route the builders
through it, the map/set analog of the pvec bulk build in #153.
phm-from-pairs partitions entries by hash and constructs the bin/array/
collision nodes directly, with the same bin<=16 / array-node>=17 promotion
the incremental path uses — so the trie is byte-identical to one built by
phm-assoc (validated across the size and branching boundaries, including
hash collisions, duplicate keys and the nil key). persistent! map/set and
the set constructor bulk-build; into {} keeps the small-scalar-map-stays-a
-struct rule via bulk-map-from-pairs; frequencies/group-by switch to the
canonical transient form and ride the fast persistent!.
50k A/B: into {} 704->270ms, frequencies 582->160, set 615->241,
into #{} 702->240, group-by 1358->919 (bound on persistent vector conj).
Gate: conformance x3, full suite (4718 >= baseline), new maps/sets bulk
boundary specs.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method reads its fields through the generic guarded keyword lookup
because the method's `this` param is untyped. defrecord now hints `this` with
the record type, the per-form inference seeds ^Record-hinted params (the
:fn branch previously typed all params :any — only the whole-program path
seeded phints), and run-passes feeds the inference the record shapes. So a
hinted param's field reads bare-index instead of going through the :jolt/type
tag guard.
This needed a with-meta fix: (with-meta sym ..) returned a proto'd table, so
symbol? was false and the macro-attached hint broke fn destructuring. Symbols
now carry metadata in-place in their struct (matching how the reader attaches
^hint), keeping symbol? true, as in Clojure.
Modest on dispatch (~3-5%): the field read is a small fraction of a dispatch;
the machinery (record-tag + protocol lookup + wrapper) dominates, which is the
inline-cache target (jolt-ez5h). But it's a correctness fix and lets any
^Record-hinted code — not just methods — drop the field-read guard per-form,
not only under whole-program.
Co-authored-by: Yogthos <yogthos@gmail.com>
Sorted collections were a sorted VECTOR — insert-at = (into (conj (subvec es
0 i) x) (subvec es i)) is O(n) per assoc with a large constant, so building was
O(n^2): 2000 entries took 55.6s.
Replace the rep with a red-black tree (assoc/dissoc/get/contains O(log n)),
ported from the ClojureScript PersistentTreeMap (cljs.core: tree-map-add /
balance-left / balance-right / tree-map-append / balance-*-del). This tier (25)
loads before 30-macros so deftype isn't available; a node is a plain vector
[color k v left right] and cljs's BlackNode/RedNode methods become functions —
the algorithm is unchanged. A sorted-set stores elements as keys with a nil
value; its ops project the key.
The seed read the old :entries vector directly for equality/printing; route
those through a new :entries op that materializes ascending from the tree
(core_types/sorted-entries-arr + main.janet's printer).
2000 sorted-map assocs: 55.6s -> 0.98s (57x); now O(log n) (per-op cost flat
from n=2000 to 10000). Correctness in test/integration/sorted-rbtree-test.janet
(shuffled insert ordering, delete rebalancing, custom comparator, comparator
lookup, subseq, count); sorted specs + full gate green. (key/val on sorted
entries stays a pre-existing gap — entries are pvecs not host tuples; jolt-jk23.)
Co-authored-by: Yogthos <yogthos@gmail.com>
A ^Record param hint was applied only at the final re-emit (reinfer-def), not
during the inter-procedural fixpoint. So a hinted param with no callers stayed
:any while inference ran, and a field read off it (e.g. (:origin ^Ray r)) never
told a non-inlined callee that its arg is a Vec3 — the callee's params stayed
unproven and its field reads kept the dynamic guard.
Seed declared hints as a param-type floor in the fixpoint: phint-seed (passes/
types) resolves an arity's :phints to positional record types via the
record-shapes registry, and infer-unit! initializes each fn's fresh param slots
from them instead of nil. A fixed declared type can't poison the least-fixpoint
the way an early-iteration :any would, and a hinted param now propagates its
(and its field reads') types to its callees during inference.
Scope: this closes the hinted-propagation gap. It does NOT help the ray tracer,
which uses zero ^-hinted params (only hinted fields) — its remaining type gap is
unhinted record-param inference on recursive/non-inlined hot fns, and per the
jolt-15jq A/B it's allocation-bound regardless (jolt-8flj). Tracked on the bead.
Co-authored-by: Yogthos <yogthos@gmail.com>
scalar-replace already folds non-escaping const-key map literals
((:k {:k a ..}) -> a, and drops a let-bound map that doesn't escape).
Extend the same fold to record constructors: a (->Rec a b c) is a
positional struct whose declared field order lives in the record-shapes
registry, so a field read on a non-escaping ctor folds to the matching
positional arg and the allocation disappears.
Direct form (:field (->Rec ..)) and the let-bound form both handled,
threaded through run-passes via a per-unit shape registry (new
jolt.host/record-shapes accessor). Soundness: ctor args must be pure
(duplicated/discarded like map vals), arg count must equal the field
count, and only declared-field reads fold — a record answers the virtual
:jolt/deftype key with its type tag and any other key with nil, neither
of which is a positional arg, so those keep the allocation. pure? now
treats a record ctor of pure args as pure, so nested records (a Ray
holding a Vec3) fold bottom-up.
Allocation-bound microbench (non-escaping record built + field-read in a
hot loop): 69.6s -> 2.4s, landing on the no-record arithmetic baseline.
The ray tracer is unchanged — its vec3 results escape (returned/stored
each op), so they genuinely allocate; that's a separate problem.
Co-authored-by: Yogthos <yogthos@gmail.com>
cap truncates a deep type's field VALUES to :any so the inter-procedural
fixpoint stays finite, but it rebuilt the struct via mk-struct and dropped the
record :type tag along the way. The tag is identity — independent of field
depth — so a record stored in a deep container (a Sphere in a world vector, a
material on a hit) degraded to a plain struct, and devirtualization (jolt-41m)
and record? folding silently stopped firing on it.
Preserve :type alongside :shape when capping. Verified: a protocol call on a
record read out of a vector now devirtualizes (the call node gets :devirt-type,
which needs the receiver's record type). Sound — the tag stays accurate; only
field values below the depth cap are truncated.
No measurable wall-clock change on its own (jolt's protocol dispatch is already
cheap), but it restores the record fast path / devirt / record?-folding on
records-in-containers, and unblocks downstream work that keys off record types.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the collection-type inference proves an argument's type, number?/
string?/keyword?/record?/nil?/some? fold to a compile-time boolean. A
const-fold now runs after inference so a folded predicate propagates and
collapses any if it gates to the taken branch.
Sound by construction: only a provable answer folds, and only when the
argument is side-effect-free (a const or local) so dropping its evaluation
is a no-op. Unknown types (:any/:truthy) and impure args keep the call.
vector?/set?/map? are left out — the :vec tag conflates a real vector with
a range/seq, so vector? could be wrong.
50M-iter loop, same shape isolated with a carry-only control: number? call+
branch 5080ms, predicate folded 1365ms — matching the 1417ms control floor,
so the 3.7x is entirely the eliminated call+branch.
Co-authored-by: Yogthos <yogthos@gmail.com>
A deftype field tagged ^:unsynchronized-mutable / ^:volatile-mutable is set!-able,
but under direct-link immutable records are shape-rec tuples, so set! errored
("Can't set! field on non-deftype: tuple").
A deftype with any mutable field now opts out of the shape-rec layout and uses
the existing :jolt/deftype table form regardless of :shapes? — set! already
mutates that form and field reads route through the tagged-table path. Such a
type is also not registered as a shape, so the inference never emits a bare-index
read against the table. Immutable deftypes/records keep the fast shape-rec.
deftype extracts per-field mutability from the field metadata and passes it to
make-deftype-ctor, which picks the representation at ctor-build time.
Co-authored-by: Yogthos <yogthos@gmail.com>
analyze-try assoc'd :catch-sym/:catch-body/:finally nil-when-absent, so a try
with no catch (or no finally) carried a nil-valued key — which makes the node a
phm in jolt's map representation and forces the back end to densify it
(norm-node) before reading :op. That's the map-nil-representation trap Phase 2
already cleaned up for def/fn/arity nodes. Add those keys only when the clause is
present, matching the arity :rest discipline; a try node stays a fast struct.
Behavior-invisible: emit-try reads each key with a nil-safe (node :k) and gates
on it, so an absent key and a present-nil key are indistinguishable to every
consumer. Adds ir-try-shape-test asserting the node shape across all four
try/catch/finally combinations plus end-to-end eval.
Note on scope: the plan's "delete the defensive norm-node calls" is NOT done — it
can't be. {:op :const :val nil} (e.g. (def x nil)) and nil map keys are
inherently phm, so the emit-dispatch norm-node guards a real case, not a
present-or-absent artifact. This PR removes a source of gratuitous phm nodes
rather than the densification itself. Full gate green.
Six bottom-up IR rewrites (const-fold, inline-node, subst, flatten-lets,
subst-lookup, scalar-replace) each hand-listed every op's child positions —
~250 lines of identical "recurse children, rebuild" arms that had to be kept in
sync whenever an op was added. Extract one map-ir-children into ir.clj that
knows each op's child layout; each walk keeps only its genuine specials
(const-fold's invoke/if, inline-node's invoke, subst's local/let alpha-rename,
scalar-replace's invoke/let folds) and delegates the rest.
The combinator is total over the op set, so the walks are now total too: a
couple soundly gain coverage they previously skipped (const-fold now folds
inside :try; subst-lookup now recurses :def inits, which fixes a latent dangling
ref where a dropped const-key-map binding was referenced inside a def). These
are sound — all six are result-preserving optimizations — and 3-mode conformance
+ fixpoint confirm identical program behavior.
map-ir-children is shape-preserving for :try (recurses :catch-body/:finally only
when present, never assoc's nil) so it can't turn a struct node into a phm.
Written with cond/get only, matching the passes' tier, so no new load-order dep.
Predicates (body-closed?/pure?/local-escapes?), the type-threading infer, and the
Janet backend emit stay as-is: their conservative :else defaults / [type node]
threading / host language don't fit a node-rebuilding combinator.
Adds ir-passes-test coverage for folding reaching fn/loop/try bodies. Full gate
green (conformance x3, suite >=4695/88, fixpoint stage1==2==3, inline-sra + devirt).
passes.clj was a 1486-line grab-bag mixing three weakly-coupled concerns. Split
along the clusters the review mapped (only run-passes + the dirty flag were
shared):
jolt.passes.fold const-fold + the shared scalar-const? predicate (base)
jolt.passes.inline inline + flatten-lets + scalar-replace
jolt.passes.types collection-type inference + success checker + driver API
jolt.passes façade: run-passes + :refer re-exports of the driver fns
the back end looks up by name
scalar-const? was used by both the inline pass and the inference walk, so it
moves to fold (the base layer) and both refer it. The check-mode state stays
private to jolt.passes.types behind a new run-inference fn; run-passes calls it.
build-compiler! loads the three in dependency order before the façade, mirroring
the existing jolt.ir -> jolt.analyzer bootstrap. No behavior change. Also fixed
the stale ns docstring that listed four passes and omitted the type system.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
* Protocol/interop fixes to run metosin/malli
Bringing up malli (schema validation) surfaced a batch of protocol and host-interop
gaps. m/validate now works across the schema vocabulary (predicates, :map incl.
nested/optional, :vector, :tuple, :enum, :maybe, :and, bounded int/string).
- extend-type and reify now accept MULTIPLE protocols in one form (each bare
symbol switches the current protocol). reify records every protocol it
implements, so instance?/satisfies? recognise all of them.
- Protocol method params support destructuring: reify/extend-type/deftype/
defrecord emit (fn ...) (which desugars patterns) instead of raw fn*.
- instance? of a PROTOCOL works like satisfies? for reify/record instances,
matching short names across qualified/bare protocol references.
- @x reads as the qualified clojure.core/deref, so it still derefs where a ns
excludes and rebinds deref (malli does). Updated reader-test + the reader
spec/grammar (S11, deref rule).
- Java collection interop on jolt collections: .nth/.count/.valAt/.get/.seq/
.containsKey route to the clojure.core equivalent (1-arg and 0-arg paths).
- java.util.HashMap capacity/load-factor constructors + .putAll.
- A class used as a value resolves to its instances' type, so Pattern -> the
regex type (malli keys class-schemas by it).
- Shims for malli's load path: LazilyPersistentVector/createOwning and
PersistentArrayMap/createWithCheck statics.
m/explain not yet working (jolt-fjb1). Full gate green.
* satisfies? recognizes reify, consistent with instance?
A reify's protocol methods are instance-local, so they aren't in the global type
registry that type-satisfies? consults — satisfies? returned false for a reify
even when it implemented the protocol. Check the protocols the reify records on
itself (the same :jolt/protocols list instance? uses), matching short names like
instance? does. Covers single- and multi-protocol reify.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add architecture refactor plan
Synthesizes a six-part architectural review into phased, gate-validated cleanup
work. Targets LLM-maintainability: one home per feature, no god-files, explicit
checked contracts, no copy-paste dispatch. No code changes yet — the plan only.
* Refactor phase 0: dead code + isolated bugs
Pure cleanup ahead of the structural phases (docs/architecture-refactor-plan.md).
No behavior change except the two bug fixes, which are covered by a regression row.
Dead code (all verified zero-reference or overridden):
- core-resolve / core-satisfies? / core-type->str seed stubs + bindings —
resolve and satisfies? are interned by install-stateful-fns! (the seed copies
were shadowed); type->str was an inert SCI stub with no callers.
- find defined twice in 20-coll.clj; the dead copy returned a plain vector
(wrong — the live def at :787 returns a real map-entry) with a comment that
contradicted it.
- mark-hint (passes.clj), phs-to-struct (phm), shape-vals / ns-imports-fn
(types) — unreferenced.
- redundant local pad2 in javatime (module-level one already in scope).
Bugs:
- File.toURL stored :url but every :jolt/url method reads :spec, so a URL from
(.toURL file) returned nil from all its methods. Now stores :spec (+ spec row).
- pl-rest had a no-op (if (plist? r) r r); collapsed to r.
- :map-shapes? was missing from the deps-image cache key — two runs differing
only in map-shapes could reuse each other's image.
Also dropped read-quote's unused pos param. Full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
A ^RecordType hint only resolved against the current namespace's ctor key, so a
hint naming a record defined in another namespace degraded to :any. That made a
decomposed multi-namespace program much slower than the monolith: per-namespace
inference can't see a record param's callers in other namespaces, and the
declared hint that could have typed it was dropped.
Resolution now works cross-namespace, for both record FIELD hints (defrecord)
and fn PARAM hints, in both spellings — ^Vec3 where the type is referred and
^v/Vec3 where the namespace is aliased:
- reader keeps a tag's namespace qualifier (^t/Ray -> "t/Ray", was "Ray").
- make-deftype-ctor-impl indexes each ctor closure by value; record-hint-ctor-key
resolves a hint name against the COMPILE ns (referred names live there; aliases
resolve through it) and maps the type var's root back to its home ctor key.
Using the ctor value, not the var's :ns, is what makes :refer work — :refer
re-interns a fresh var whose :ns is the referring ns.
- the analyzer captures record param hints as arity :phints [name ctor-key];
reinfer-def seeds those param types, so a record param is typed even with no
inferred caller — the open-world / cross-ns case.
Effect on the multi-namespace ray tracer: per-ns compile 30.4s -> 7.9s with
param hints, matching whole-program (8.1s) and the single-ns monolith (8.3s).
cross-ns-hints-test covers field + param hints, refer + as, and the reader tag.
A record field can carry a type hint — ^Vec3 (a defined record type) or ^:num —
and the inference now resolves it so reading the field back yields that exact type
instead of :any. A Vec3 stored in a Ray field reads out as Vec3, so the vec ops on
field-read values prove their reads (bare-index). This is Stalin's per-slot type
sets, but DECLARED rather than inferred: the exact shape is known up front.
- deftype captures each field's :tag / :num metadata (was stripped) and passes it
to make-deftype-ctor; the ctor registers per-field tags, resolving a record-type
hint to its ctor-key (same-ns) so the inference can look it up directly.
- call-ret-type builds a record's struct type with field types resolved from the
hints, recursing into nested record types (depth-bounded for self/cyclic types).
Measured: a nested-record read loop (:r (:origin ray)) runs 1.3s with ^Vec3 hints
vs 7.1s without — 5.5x. This is the lever the ray tracer needed (vecs flow through
container fields); records without it read back as :any and stay unproven.
A protocol method call compiles to (protocol-dispatch proto method this rest) — a
runtime registry walk (type-tag -> proto -> method) on every call, ~19x a direct
call. When the inference proves the receiver (arg 0) is a known record type, the
call now resolves to a DIRECT method call at compile time, skipping the registry.
- defprotocol registers each method's var-key 'ns/method' -> [proto method] (a
ctx-capturing register-protocol-methods! emitted into the do-block); infer-unit!
feeds it to the inference via a box (like record-shapes).
- the record-ctor return type carries :type (the record tag) so the inference
knows the receiver type; the :else invoke case annotates a protocol call whose
arg0 has a known :type with :devirt-{type,proto,method}.
- emit-invoke resolves the impl via find-protocol-method at emit time and emits a
direct call to the embedded impl fn value. Unknown/polymorphic receivers (no
proven :type) fall back to the dispatch path unchanged.
Measured: removes the dispatch overhead (14.7s -> 9.3s on a 10M-call loop); the
remaining cost is the method body itself (non-inlined, unproven reads) — inlining
the resolved method is the follow-up (jolt-t6r) toward direct-call speed.
Sound under the closed-world assumption direct-linking already makes (the impl is
resolved + embedded at compile time). Adds devirt-test (subprocess: dispatched ==
devirtualized across polymorphic dispatch, unknown-receiver fallback, and
heterogeneous collections). Stalin's compile-call/callee-environment is the model.
Records (defrecord/deftype) are now shape-recs in a direct-linking unit by
default — no JOLT_SHAPE flag. A record's shape is DECLARED, so the inference
proves field reads by a lookup, not fragile shape inference, and they bare-index.
Result: ~1.4x faster than the :jolt/deftype table form on a record-heavy loop
(3.9s vs 5.5s), driven by cheaper construction + proven bare-index reads.
Two gates now:
- :shapes? — shape-recs active; records use declared-shape layout + bare
index reads. On with direct-linking (where the inference runs).
- :map-shapes? — also shape generic const-key maps. Opt-in (JOLT_SHAPE), because
shaping maps net-loses on unproven reads (measured). Records win.
- call-ret-type types a record ctor (->Name) as a struct of its declared shape,
fed from a ctx-env registry populated at deftype; field reads on the result
bare-index. (set-record-shapes!/set-map-shapes! wired through infer-unit!.)
- sidx reads the field's position from the :shape vector AS-IS (declared order
for records, str-sorted for map literals) — no re-sort — so any field order
bare-indexes correctly. The map :map case only sets :shape under :map-shapes?.
- record-shape-for interns the descriptor per (type, fields), not per type: a
record redefined with different fields now gets a fresh descriptor instead of a
stale one (fixes redef descriptor staleness; old instances stay valid).
Adds record-declared-shape-test (declared-order reads, incl. non-alphabetical
fields, through fn boundaries + protocol method bodies). Known pre-existing edge
case filed as jolt-wf4 (direct (:f (->R …)) read returns nil after a record is
redefined with different fields; let-bound read works; repros without shapes).
The inference dropped the complete :shape whenever it rebuilt a struct type
(cap) or joined two (join-t/merge-fields), so a vec3 retrieved from a container
or a fn param typed across call sites lost its layout and every field read fell
to the slow descriptor path. Two fixes:
- cap preserves :shape: capping truncates field VALUES below the depth limit but
never the key SET, so the layout is still complete. It also recurses into
fields, so a shaped value nested in a container (a vec3 inside a hit-info)
keeps its own :shape — which is what lets (:r (:normal hit-info)) bare-index.
- join-t preserves :shape when both sides are the SAME complete shape (the
merged struct has the same keys); different shapes still drop it. This carries
the shape through if-joins and the inter-procedural fixpoint's call-site joins.
Result: the ray tracer goes from 22s (R1, correct-but-descriptor-path) to 4.36s
— 2.7x FASTER than the 11.7s no-shape baseline, and ~3x the JVM (was 8.5x), with
byte-identical output. The compounding of cheaper tuple construction plus
bare-index reads across the whole render far exceeds the per-op estimate.
Gate green flag-off, suite 4718, default-path bench even, transparency intact.
Removes the {:r :g :b} hardcoding. ANY constant key set is now a shape:
- inference: a struct type from a map LITERAL carries :shape (its canonical
str-sorted key vector — completeness); joins/access-inferred structs lack
it, so they never get a bare index. The literal node and lookup subjects
carry the shape; the back end derives the index from it.
- backend: emit-map turns any shape-tagged const-key map into a shape tuple;
emit-kw-lookup reads the field by bare index when the complete shape is
proven, else by the value's own descriptor (so a shape-rec whose :shape was
dropped by a join still reads correctly).
- runtime: core-get and core-assoc handle shape-recs.
Status: CORRECT for direct field access, container round-trips, and assoc
(minimal repros pass). NOT yet complete — the full ray tracer still hits an
uncovered path (a shape-rec reaching a map op without coverage: keys/vals/
count/seq/equality/print/jolt-call/dissoc/contains?/the interpreter's
coll-lookup all still need shape-rec branches). And the perf win needs
COMPLETENESS PRESERVATION through joins/containers (merge-fields/cap drop
:shape today, so nested vec3 access falls to the descriptor path, slower than
a struct get) — without it the general version is slower than the vec3
prototype.
All behind JOLT_SHAPE (off by default). Gate green with the flag off, suite
4718. This preserves the general design; the transparency layer + completeness
preservation are the remaining multi-session work.
Validated prototype of the hidden-class object-model change. A vec3-shaped
{:r :g :b} map literal is represented as a cheap Janet tuple [shape vb vg vr]
instead of a struct (~2x cheaper to construct); a lookup on a value the
inference PROVES is the shape reads by bare index with no runtime check.
Result on the ray tracer (direct-link): 12.3s -> 10.7s (~13% faster), with
byte-identical pixel output. The shape value flows transparently through
hit-info/ray/material containers and the colors vector; core-get handles it
(inline check, no fn call) so an unspecialized access is still correct.
Key lessons baked in: the lookup MUST compile to a bare index (a runtime
shape check, even inlined, taxed every field read and made it 2.5-3.4x
SLOWER) — so the inference gained a :shape hint (struct type with keys
exactly {:r :g :b}) that the back end turns into (in m idx). The descriptor
is quoted when embedded (its keys are a parens tuple Janet would otherwise
try to CALL).
All behind JOLT_SHAPE (off by default). Gate green, suite 4718, default-path
bench even. Scoped to the one shape; NOT yet sound in general (assumes every
vec3-shaped value is a shape-rec, true under the flag for the ray tracer) nor
fully transparent (only core-get + the inlined lookup; jolt-call/equality/
print/keys not yet covered). Those are the next steps toward a real feature.
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
File API (jolt-hjw): io/file and (File. …) build a tagged :jolt/file value
(instance? File true) with a full method surface (isFile/isDirectory/exists/
getName/getPath/getAbsolutePath/listFiles/toPath/delete/createNewFile/…) backed
by os/ and file/. file-seq is File-aware (leaves are File values). str/slurp/spit
coerce :jolt/file to its path. ClassLoader/getSystemClassLoader + a classloader
stub whose getResource returns nil degrade migratus's classpath lookup to the
filesystem. java.nio.file Path/FileSystem/PathMatcher are shimmed just enough for
script-excluded?'s glob (recursive * / ? matcher).
Three bugs found getting migratus's migration discovery to work:
- (assoc nil k v) returned a raw janet table, not a map, so assoc-in built tables
that count/seq rejected. Now returns an immutable map.
- methods/get-method resolved the multimethod symbol at runtime in the current
ns, so a bare multifn ref in its defining ns saw an empty table once defmethods
lived elsewhere. Now they take the multimethod VALUE and recover the var via a
registry (Clojure semantics).
- defmulti now drops a leading docstring/attr-map (migratus's multimethods carry
docstrings) instead of treating the docstring as the dispatch fn.
Conformance 335/335 x3, clojure-test-suite at baseline.
The analyzer always took (nth items 2) as the value, so (def x "doc" 42)
bound x to the docstring and dropped 42. Now it mirrors the interpreter:
when there are 4+ items and item 2 is a string, item 2 is the docstring
(attached as :doc meta) and item 3 is the value. Conformance 335/335 x3.
For the migratus next.jdbc shim (jolt-0z5):
- core.janet: __jdbc-wrap-conn / __jdbc-conn-raw / __jdbc-make-stmt builtins.
A connection is a tagged wrapper over a jdbc.core conn carrying a clj :exec
callback so the host Statement.executeBatch runs SQL without a janet->clj call.
- javatime.janet: tagged-methods for :jolt/jdbc-conn (setAutoCommit/isClosed/
close/getMetaData), :jolt/jdbc-meta (getDatabaseProductName), :jolt/jdbc-stmt
(addBatch/executeBatch/close); java.sql.Timestamp ctor -> millis.
- evaluator.janet: instance? case for Connection/java.sql.Connection so
migratus's do-commands runs SQL through its Connection branch.
Two defn/defmacro fixes found loading migratus.core (both rooted in the reader
representing ^{:map} name metadata as a with-meta form, jolt-8w2):
- defmacro special form: unwrap a with-meta name (mirrors def), and handle the
arity-clause form (defmacro name ([params] body...)) like fn/defn — a params
vector reads as a tuple, an arity clause as a list (array).
- defn overlay: pass the bare (unwrapped) name to fn while def keeps the meta.
Conformance 335/335 x3 modes.
Two provably-wrong cases the inference already has the facts for, closing the
last RFC 0006 open question:
- Calling a non-function. At an :invoke whose callee is provably :num or :str
(the only non-callable types — keywords/maps/vectors/sets are IFn), report
"cannot call a number as a function". Default level (no closed-world: the
callee type is inferred at the call site). Covers (5 1), ("hi" 0),
((+ 1 2) :k), a let-bound number, and a var holding a number (via vtype-box
in direct-link). A union is non-callable only when every member is, so
((if c 1 :k) x) is accepted (:kw is callable). Verified zero false positives
on the ray tracer, which calls maps/keywords/vectors as fns throughout.
- Wrong arity to a user fn. The registered single-fixed-arity sig (jolt-zo1)
makes a mismatched arg count provably throw; reported under the
JOLT_TYPE_CHECK_USER opt-in (same closed-world boundary; ^:redef/variadic
skipped). Caught at compile time before the runtime arity error.
Both fold into the existing infer walk, carry :pos for file:line:col, and keep
no-false-positives. Gate green, suite 4718, conformance 335/335, runtime bench
even (compile-time only).
Checking inherently needs an inference pass (~2.6x compile as a standalone
pass). But direct-link builds ALREADY run one inference pass for
specialization (run-passes' infer-top), so checking can ride along: set a
check-mode flag, turn checking? on during that existing pass, and collect
the diagnostics after — ~2% overhead measured on the ray tracer, vs 2.6x
for the separate pass.
So the checker now defaults to `warn` in direct-link builds (where it's
nearly free) and stays OFF in plain REPL/dev builds (no inference to ride,
no forced cost — opt in with JOLT_TYPE_CHECK there). JOLT_TYPE_CHECK still
overrides in both directions (off to disable, error to escalate).
It checks the POST-optimization IR, which matches what the optimized
program actually evaluates — scalar-replace only drops provably-pure code,
an accepted opt-mode divergence, so no real error is hidden. The loaders
enable position tracking whenever checking will run (env-selected or
direct-link). type-check! (the standalone pass) stays for plain builds;
both paths share report-diags!.
cli-test pins: plain build silent, direct-link warns by default,
JOLT_TYPE_CHECK=off disables. Gate green, suite 4718, runtime bench even.
The checker ran a separate check-walk that re-inferred each argument's
subtree AND recursed into it — quadratic in expression nesting. Fold the
diagnostic emission into `infer` itself (gated by a checking? flag, off
during the optimization fixpoint): one O(n) walk that both types and
checks. Removes check-walk entirely; check-form now drives infer.
This is a cleanup and removes the deep-nesting blowup, but it does NOT make
warn-by-default cheap: on a real 360-line file the checker still adds ~2.6x
compile time (277ms -> 720ms). That cost is the structural inference pass
itself, which checking inherently requires — not redundancy. A cheap
default-on path would need either piggybacking on the inference direct-link
already runs, or a lighter scalar-only checker inference. Gate green,
type-check tests pass.
RFC 0006 error reporting wanted file:line:col but IR nodes carried no
position, so diagnostics read only "type error in <ns>: <msg>". Now:
type error /tmp/scene.clj:5:5: `inc` requires a number, but argument 1 is a string
The reader records each LIST form's absolute start offset in a table keyed
by form identity (lists are fresh arrays, never interned), gated behind a
flag the loaders enable only when JOLT_TYPE_CHECK is on — zero cost off.
Keying by identity makes positions survive macroexpansion exactly when the
user's own sub-form is spliced through, and absent for macro-synthesized
structure: a `(inc :k)` written inside `(when c ...)` reports at its own
line, never at the expansion's generated if/do.
The analyzer stamps the offset onto :invoke nodes (form-position host
contract fn); the checker carries it into each diagnostic as :pos; the
loaders stash the file's source + path on the env (save/restored across
nested requires); backend/type-check! converts offset -> line:col via the
reader's line-col and renders the RFC format. Falls back to the ns when no
position is available (synthetic forms), so it is never worse than before.
Gate green, conformance 335/335, suite 4718, runtime bench even (positions
are compile-time only; off by default).
The success checker fired only against core-fn error domains (stable, not
redefinable). This adds reporting of a call that passes a provably-wrong
type to a USER fn whose body requires otherwise — e.g. a fn that only does
arithmetic on a param, called with a string.
As check-walk sees defs it registers each non-redefinable single-fixed-arity
user fn's {:params :body} in module state (user-sig-box, accumulating across
forms like rtenv-box — a def must precede its call). At a call site (strict
mode only) the body is re-checked with ONE parameter bound to its concrete
argument type, others :any; if that produces a diagnostic the all-:any body
did not, the argument alone is provably wrong and the call is reported.
Monotonic — binding a concrete type can only add error-domain hits — so still
no false positives. A cycle guard (checking-box) terminates mutual recursion.
Gated behind JOLT_TYPE_CHECK_USER (orthogonal to the warn/error level)
because it rests on the closed-world assumption, weaker than the core-fn
case. check-form gains a strict? arity; the default path is unchanged and
user-fn code runs only when the checker is enabled. ^:redef/^:dynamic and
multi/variadic fns are not registered (their body is no stable requirement).
Gate green, suite 4718, conformance 335/335.
The success checker (RFC 0006) used to lose differing if-branches to :any
and accept the use. (inc (if c "a" :k)) typed the if as :any — sound but
imprecise, since the value is provably {:str | :kw}, every member of which
is in inc's error domain.
Adds {:union #{T...}} to the lattice: join-t forms a scalar union of
differing branches instead of collapsing to :any, capped at 4 distinct
scalars (the member space is the five scalar tags, so the lattice stays
finite and the inter-procedural fixpoint still terminates). The checker's
not-number?/not-seqable? report a union only when EVERY member is in the
error domain — any valid member accepts the call, so still no false
positives. type-name renders "a string or a keyword".
Unions are scalar-only and carry no :struct/:vec/:set key, so every
structural predicate already treats them as opaque — specialization sees
them exactly as :any and codegen is unchanged. Gate green, suite 4718,
conformance 335/335, bench even.
Reuse the structural inference from RFC 0005 as a loose type checker. It reports
a core-fn call only when an argument's inferred type is concrete and lies in
that op's throwing error domain, and accepts everything ambiguous (:any, a
union that joined to :any, :truthy). By construction it never produces a false
positive: a correct program has nothing to report even in error mode.
The curated error-domain table starts with the clearest throwing cases:
arithmetic on a provable non-number, and count/first/rest/next/seq/nth on a
provable non-seqable scalar. Lenient operations like (get 5 :k) and (:k 5),
which return nil rather than throw, are deliberately not listed.
Checking is decoupled from specialization: it runs whenever JOLT_TYPE_CHECK is
warn or error, regardless of :inline?, reading the knob at compile time so no
rebuild is needed. warn prints to stderr, error fails the form's compilation,
off (the default) skips it entirely. Core init stays clean under the flag.
jolt-y3b
