Lower host class interop on the Chez back end. The analyzer now turns a
non-var qualified ref `Class/member` into a :host-static node and a
`(Class. ...)` / `(new Class ...)` form into a :host-new node (ir.clj
gains both, with walker support). The Janet back end punts both to the
interpreter, so its behavior is unchanged (verified: dot-form, `..`
threading, shadowed `new`, and all interop still resolve via fallback).
The Chez emit lowers a value ref to host-static-ref, a call head to
host-static-call, and a constructor to host-new. host/chez/host-static.ss
is the runtime registry these resolve against — the Chez port of the
seed's class-statics / class-ctors / tagged-methods (java_base.janet +
host_io.janet), restricted to the java.lang/util/net/io surface portable
cljc code calls: Math, System (getenv/getProperty/exit/currentTimeMillis),
Long, Integer, Boolean, Character, String, Thread, Class, Pattern
(compile/quote/MULTILINE), URLEncoder/Decoder, Base64, the Number method
surface (byteValue/intValue/...), plus the StringBuilder, StringWriter,
StringReader, PushbackReader, HashMap, StringTokenizer, BigInteger,
String, MapEntry, and exception constructors. Constructed objects are
jhost records dispatched through record-method-dispatch.
Also: emit now evaluates collection-literal elements left-to-right
(emit-ordered) — Chez evaluates call args right-to-left, which had been
swapping side-effecting elements in [(read r) (read r)] and map literals.
This un-allowlisted the 6 eval-order corpus cases (the read-line trio +
the three map-construction cases). Removed `.write` from the
jolt-host-call fast-path so a StringWriter routes through dispatch.
java.time formatting, edn/read-over-readers, and slurp/with-open over
readers are deferred to a follow-up.
Corpus parity 2078 -> 2134 (floor raised), 0 new divergences; the
print-method builtin-override case is allowlisted (same multimethod gap,
newly reachable now that StringWriter constructs). emit-test 326/326,
_javastatic 51/51, conformance 355x3, full jpm test green.
host/chez/multimethods.ss implements the multimethod runtime: defmulti/defmethod
expand to defmulti-setup/defmethod-setup calls (+ get-method/methods/
remove-method/prefer-method/prefers). A jolt-multifn record carries its dispatch
fn and a jolt=-keyed method table; jolt-invoke dispatches it (exact match, then
isa?/hierarchy with prefer-method, then :default), reusing the overlay's
isa?/derive/make-hierarchy. The multifn's ns comes from a runtime chez-current-ns
(default user; the prelude load sets clojure.core for print-method/print-dup).
Two emit-side changes were needed:
- late-bind (:late-bind-unresolved? ctx flag, default OFF): defmulti expands to a
bare-symbol setup call, so the analyzer doesn't intern the name and a forward
reference '(area ...)' after '(defmulti area ...)' in one form was 'Unable to
resolve symbol'. The strict compiler punts these to the interpreter; the Chez
back end has none, so the flag lowers an unresolved symbol to a var-ref against
the compile ns (open-world -e semantics). Set only by the Chez make-ctx /
jolt-chez; the main compiler keeps strict resolution (host_iface late-bind?
defaults nil).
- a :var call head now routes through jolt-invoke, since a late-bound var can hold
a multifn (or keyword/coll IFn), not just a procedure. Transparent for
procedures; the hot self-recursive call is a :local known-proc, stays direct.
Class-based dispatch ((class x)/String) deferred (needs deftype/class subsystem).
Parity 1506 -> 1530/2497, 0 new divergences. emit-test 302/302. Full janet gate
green (the analyzer flag is off there; suite flakiness under parallel load only).
Closes the last clojure.core prelude emit gap (parse-uuid): the whole
non-macro core now lowers to Scheme (prelude reach 355/355).
A #"..." literal analyzes to a :regex IR node. The Chez back end emits
a jolt-regex value over irregex (Alex Shinn, BSD), vendored as the
vendor/irregex submodule -- a portable Scheme regex with PCRE/Java-style
string patterns and first-class Chez support. host/chez/regex.ss wraps
jolt's re-* surface over it: irregex-match -> re-matches (anchored),
irregex-search -> re-find, groups as Clojure [whole g1 ...] vectors,
re-seq as a jolt seq. re-pattern/re-matches/re-find/re-seq/regex? are
def-var!'d into clojure.core so prelude / -e code resolves them.
They stay OUT of the subset native-ops on purpose: irregex's
Unicode/property-class semantics differ from the seed's byte-PEG
approximation, so keeping them prelude-only avoids dragging
engine-difference divergences into the subset-parity corpus. The Janet
back end punts :regex to the interpreter (the seed compiles #"..." to a
Janet PEG), so the main language is unchanged.
Only two adaptations for Chez's top level: a cond-expand shim (Chez's is
library-only) and a normalizing error wrapper (silences irregex's 1-arg
error warnings). rt.ss load is ~0.18s.
emit-test 131/131 (regex literal + re-* parity vs the CLI oracle);
prelude reach 355/355; Chez subset 672/672, 0 divergences; full gate
green.
(.method target arg*) now analyzes to a :host-call IR node instead of
punting at analyze. The Chez back end lowers it to a jolt-host-call
dispatch for the methods the RT shims (.write -> port display,
.isDirectory -> file-directory?, .listFiles -> directory-list); any
other method stays out of subset (clean emit-time reject, so it can't
read as a compiled-but-broken corpus divergence). The Janet back end
punts ALL :host-call to the interpreter, same shape as letfn: compiles
on Chez, interprets on Janet, zero change to the main language.
Closes the io tier's print-method defmethods and file-seq: prelude emit
reach 348 -> 354/355 (50-io 20/20). The one remaining gap is the regex
literal in parse-uuid (needs a regex engine on Chez; deferred).
emit-test 122/122; Chez subset 672/672, 0 divergences; full gate green.
Closes the last two non-host-interop prelude emit gaps.
letfn now analyzes to a :let node flagged :letrec — the binding fns are bound
into the env together before any spec is analyzed, so siblings and self resolve.
The Chez back end lowers it to letrec*; the Janet back end punts it at emit
(its sequential let* can't express the mutual recursion — same interpreter
fallback as before, just decided at emit-ir instead of analyze).
(def x) with no init (declare) analyzes to a :def with :no-init instead of
punting. Chez reserves the var cell via declare-var! (which doesn't clobber an
existing root — (do (def x 7) (def x) x) => 7); the Janet back end still punts
to the interpreter, which interns a genuinely-unbound var.
fallback-zero-test now checks emit-ir too, not just analyze-form, so the real
compile-vs-interpret decision is what it asserts (letfn/def-no-init analyze but
the Janet back end punts them). letfn stays in must-punt with an updated note.
Prelude emit reach 342 -> 348/355 (40-lazy now 13/13); Chez subset 664 -> 672,
0 divergences; emit-test 110 -> 117. Full gate green.
jolt's catch is (catch class binding body*); the binding (3rd element) must be
a symbol. Neither the analyzer nor the interpreter validated it, so a non-symbol
binding crashed with an internal Janet error (expected integer key for array...)
and, in the interpreter, a malformed clause whose body never threw was silently
swallowed (returned the try value). Clojure rejects a non-class/non-symbol catch
clause; match that with an up-front error in analyze-try and eval-try.
Surfaced building the Chez try/throw emit. Regression rows in exceptions-spec
(runs x3 modes) plus a unit test asserting the clean message in interpret and
compile. jolt-kg6p.
The map build already used a transient map, but each bucket was rebuilt with
a persistent (conj (get ret k []) x) per element — an O(log n) trie path
rebuild + alloc each. A coarse grouping (few large buckets) was bound on that
conj, not the map build. Buckets are now native arrays (transient vectors,
O(1) push) frozen once; distinct keys are tracked in a side vector so the
buckets freeze in place with no second map rebuild. A bucket's first element
stays a cheap persistent [x] and only promotes to a transient on the second,
so an all-singletons grouping pays no transient alloc.
coarse (10/100 buckets, 50k): ~313ms -> ~125ms (~2.5x)
2 buckets (50k): ~322ms -> ~129ms (~2.5x)
all-unique (50k): ~949ms -> ~892ms (no regression)
Surfaced a latent bug: canon-key returns nil for a nil key and Janet tables
drop a nil key, so the canon-keyed transient map silently lost a nil-key
entry — group-by/frequencies/assoc!/into{} dropped the whole nil bucket
((group-by identity [nil nil 1]) gave {1 [1]}, not {nil [nil nil], 1 [1]}).
Route nil through a sentinel (tbl-key) at the transient-map keying sites;
persistent!/count/dissoc! work unchanged since the real [nil v] pair is kept
as the stored value, and phm already has its own has-nil slot. The transient
set has the analogous bug (needs phs nil support) — filed separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
into {}, frequencies, group-by, set, into #{} and persistent! all built
their result by folding an immutable assoc/conj per element — each call
rebuilt the O(log32 n) trie path and allocated a fresh wrapper. Add a
one-pass bottom-up HAMT builder (phm-from-pairs) and route the builders
through it, the map/set analog of the pvec bulk build in #153.
phm-from-pairs partitions entries by hash and constructs the bin/array/
collision nodes directly, with the same bin<=16 / array-node>=17 promotion
the incremental path uses — so the trie is byte-identical to one built by
phm-assoc (validated across the size and branching boundaries, including
hash collisions, duplicate keys and the nil key). persistent! map/set and
the set constructor bulk-build; into {} keeps the small-scalar-map-stays-a
-struct rule via bulk-map-from-pairs; frequencies/group-by switch to the
canonical transient form and ride the fast persistent!.
50k A/B: into {} 704->270ms, frequencies 582->160, set 615->241,
into #{} 702->240, group-by 1358->919 (bound on persistent vector conj).
Gate: conformance x3, full suite (4718 >= baseline), new maps/sets bulk
boundary specs.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method reads its fields through the generic guarded keyword lookup
because the method's `this` param is untyped. defrecord now hints `this` with
the record type, the per-form inference seeds ^Record-hinted params (the
:fn branch previously typed all params :any — only the whole-program path
seeded phints), and run-passes feeds the inference the record shapes. So a
hinted param's field reads bare-index instead of going through the :jolt/type
tag guard.
This needed a with-meta fix: (with-meta sym ..) returned a proto'd table, so
symbol? was false and the macro-attached hint broke fn destructuring. Symbols
now carry metadata in-place in their struct (matching how the reader attaches
^hint), keeping symbol? true, as in Clojure.
Modest on dispatch (~3-5%): the field read is a small fraction of a dispatch;
the machinery (record-tag + protocol lookup + wrapper) dominates, which is the
inline-cache target (jolt-ez5h). But it's a correctness fix and lets any
^Record-hinted code — not just methods — drop the field-read guard per-form,
not only under whole-program.
Co-authored-by: Yogthos <yogthos@gmail.com>
Sorted collections were a sorted VECTOR — insert-at = (into (conj (subvec es
0 i) x) (subvec es i)) is O(n) per assoc with a large constant, so building was
O(n^2): 2000 entries took 55.6s.
Replace the rep with a red-black tree (assoc/dissoc/get/contains O(log n)),
ported from the ClojureScript PersistentTreeMap (cljs.core: tree-map-add /
balance-left / balance-right / tree-map-append / balance-*-del). This tier (25)
loads before 30-macros so deftype isn't available; a node is a plain vector
[color k v left right] and cljs's BlackNode/RedNode methods become functions —
the algorithm is unchanged. A sorted-set stores elements as keys with a nil
value; its ops project the key.
The seed read the old :entries vector directly for equality/printing; route
those through a new :entries op that materializes ascending from the tree
(core_types/sorted-entries-arr + main.janet's printer).
2000 sorted-map assocs: 55.6s -> 0.98s (57x); now O(log n) (per-op cost flat
from n=2000 to 10000). Correctness in test/integration/sorted-rbtree-test.janet
(shuffled insert ordering, delete rebalancing, custom comparator, comparator
lookup, subseq, count); sorted specs + full gate green. (key/val on sorted
entries stays a pre-existing gap — entries are pvecs not host tuples; jolt-jk23.)
Co-authored-by: Yogthos <yogthos@gmail.com>
A ^Record param hint was applied only at the final re-emit (reinfer-def), not
during the inter-procedural fixpoint. So a hinted param with no callers stayed
:any while inference ran, and a field read off it (e.g. (:origin ^Ray r)) never
told a non-inlined callee that its arg is a Vec3 — the callee's params stayed
unproven and its field reads kept the dynamic guard.
Seed declared hints as a param-type floor in the fixpoint: phint-seed (passes/
types) resolves an arity's :phints to positional record types via the
record-shapes registry, and infer-unit! initializes each fn's fresh param slots
from them instead of nil. A fixed declared type can't poison the least-fixpoint
the way an early-iteration :any would, and a hinted param now propagates its
(and its field reads') types to its callees during inference.
Scope: this closes the hinted-propagation gap. It does NOT help the ray tracer,
which uses zero ^-hinted params (only hinted fields) — its remaining type gap is
unhinted record-param inference on recursive/non-inlined hot fns, and per the
jolt-15jq A/B it's allocation-bound regardless (jolt-8flj). Tracked on the bead.
Co-authored-by: Yogthos <yogthos@gmail.com>
scalar-replace already folds non-escaping const-key map literals
((:k {:k a ..}) -> a, and drops a let-bound map that doesn't escape).
Extend the same fold to record constructors: a (->Rec a b c) is a
positional struct whose declared field order lives in the record-shapes
registry, so a field read on a non-escaping ctor folds to the matching
positional arg and the allocation disappears.
Direct form (:field (->Rec ..)) and the let-bound form both handled,
threaded through run-passes via a per-unit shape registry (new
jolt.host/record-shapes accessor). Soundness: ctor args must be pure
(duplicated/discarded like map vals), arg count must equal the field
count, and only declared-field reads fold — a record answers the virtual
:jolt/deftype key with its type tag and any other key with nil, neither
of which is a positional arg, so those keep the allocation. pure? now
treats a record ctor of pure args as pure, so nested records (a Ray
holding a Vec3) fold bottom-up.
Allocation-bound microbench (non-escaping record built + field-read in a
hot loop): 69.6s -> 2.4s, landing on the no-record arithmetic baseline.
The ray tracer is unchanged — its vec3 results escape (returned/stored
each op), so they genuinely allocate; that's a separate problem.
Co-authored-by: Yogthos <yogthos@gmail.com>
cap truncates a deep type's field VALUES to :any so the inter-procedural
fixpoint stays finite, but it rebuilt the struct via mk-struct and dropped the
record :type tag along the way. The tag is identity — independent of field
depth — so a record stored in a deep container (a Sphere in a world vector, a
material on a hit) degraded to a plain struct, and devirtualization (jolt-41m)
and record? folding silently stopped firing on it.
Preserve :type alongside :shape when capping. Verified: a protocol call on a
record read out of a vector now devirtualizes (the call node gets :devirt-type,
which needs the receiver's record type). Sound — the tag stays accurate; only
field values below the depth cap are truncated.
No measurable wall-clock change on its own (jolt's protocol dispatch is already
cheap), but it restores the record fast path / devirt / record?-folding on
records-in-containers, and unblocks downstream work that keys off record types.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the collection-type inference proves an argument's type, number?/
string?/keyword?/record?/nil?/some? fold to a compile-time boolean. A
const-fold now runs after inference so a folded predicate propagates and
collapses any if it gates to the taken branch.
Sound by construction: only a provable answer folds, and only when the
argument is side-effect-free (a const or local) so dropping its evaluation
is a no-op. Unknown types (:any/:truthy) and impure args keep the call.
vector?/set?/map? are left out — the :vec tag conflates a real vector with
a range/seq, so vector? could be wrong.
50M-iter loop, same shape isolated with a carry-only control: number? call+
branch 5080ms, predicate folded 1365ms — matching the 1417ms control floor,
so the 3.7x is entirely the eliminated call+branch.
Co-authored-by: Yogthos <yogthos@gmail.com>
A deftype field tagged ^:unsynchronized-mutable / ^:volatile-mutable is set!-able,
but under direct-link immutable records are shape-rec tuples, so set! errored
("Can't set! field on non-deftype: tuple").
A deftype with any mutable field now opts out of the shape-rec layout and uses
the existing :jolt/deftype table form regardless of :shapes? — set! already
mutates that form and field reads route through the tagged-table path. Such a
type is also not registered as a shape, so the inference never emits a bare-index
read against the table. Immutable deftypes/records keep the fast shape-rec.
deftype extracts per-field mutability from the field metadata and passes it to
make-deftype-ctor, which picks the representation at ctor-build time.
Co-authored-by: Yogthos <yogthos@gmail.com>
analyze-try assoc'd :catch-sym/:catch-body/:finally nil-when-absent, so a try
with no catch (or no finally) carried a nil-valued key — which makes the node a
phm in jolt's map representation and forces the back end to densify it
(norm-node) before reading :op. That's the map-nil-representation trap Phase 2
already cleaned up for def/fn/arity nodes. Add those keys only when the clause is
present, matching the arity :rest discipline; a try node stays a fast struct.
Behavior-invisible: emit-try reads each key with a nil-safe (node :k) and gates
on it, so an absent key and a present-nil key are indistinguishable to every
consumer. Adds ir-try-shape-test asserting the node shape across all four
try/catch/finally combinations plus end-to-end eval.
Note on scope: the plan's "delete the defensive norm-node calls" is NOT done — it
can't be. {:op :const :val nil} (e.g. (def x nil)) and nil map keys are
inherently phm, so the emit-dispatch norm-node guards a real case, not a
present-or-absent artifact. This PR removes a source of gratuitous phm nodes
rather than the densification itself. Full gate green.
Six bottom-up IR rewrites (const-fold, inline-node, subst, flatten-lets,
subst-lookup, scalar-replace) each hand-listed every op's child positions —
~250 lines of identical "recurse children, rebuild" arms that had to be kept in
sync whenever an op was added. Extract one map-ir-children into ir.clj that
knows each op's child layout; each walk keeps only its genuine specials
(const-fold's invoke/if, inline-node's invoke, subst's local/let alpha-rename,
scalar-replace's invoke/let folds) and delegates the rest.
The combinator is total over the op set, so the walks are now total too: a
couple soundly gain coverage they previously skipped (const-fold now folds
inside :try; subst-lookup now recurses :def inits, which fixes a latent dangling
ref where a dropped const-key-map binding was referenced inside a def). These
are sound — all six are result-preserving optimizations — and 3-mode conformance
+ fixpoint confirm identical program behavior.
map-ir-children is shape-preserving for :try (recurses :catch-body/:finally only
when present, never assoc's nil) so it can't turn a struct node into a phm.
Written with cond/get only, matching the passes' tier, so no new load-order dep.
Predicates (body-closed?/pure?/local-escapes?), the type-threading infer, and the
Janet backend emit stay as-is: their conservative :else defaults / [type node]
threading / host language don't fit a node-rebuilding combinator.
Adds ir-passes-test coverage for folding reaching fn/loop/try bodies. Full gate
green (conformance x3, suite >=4695/88, fixpoint stage1==2==3, inline-sra + devirt).
passes.clj was a 1486-line grab-bag mixing three weakly-coupled concerns. Split
along the clusters the review mapped (only run-passes + the dirty flag were
shared):
jolt.passes.fold const-fold + the shared scalar-const? predicate (base)
jolt.passes.inline inline + flatten-lets + scalar-replace
jolt.passes.types collection-type inference + success checker + driver API
jolt.passes façade: run-passes + :refer re-exports of the driver fns
the back end looks up by name
scalar-const? was used by both the inline pass and the inference walk, so it
moves to fold (the base layer) and both refer it. The check-mode state stays
private to jolt.passes.types behind a new run-inference fn; run-passes calls it.
build-compiler! loads the three in dependency order before the façade, mirroring
the existing jolt.ir -> jolt.analyzer bootstrap. No behavior change. Also fixed
the stale ns docstring that listed four passes and omitted the type system.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
* Protocol/interop fixes to run metosin/malli
Bringing up malli (schema validation) surfaced a batch of protocol and host-interop
gaps. m/validate now works across the schema vocabulary (predicates, :map incl.
nested/optional, :vector, :tuple, :enum, :maybe, :and, bounded int/string).
- extend-type and reify now accept MULTIPLE protocols in one form (each bare
symbol switches the current protocol). reify records every protocol it
implements, so instance?/satisfies? recognise all of them.
- Protocol method params support destructuring: reify/extend-type/deftype/
defrecord emit (fn ...) (which desugars patterns) instead of raw fn*.
- instance? of a PROTOCOL works like satisfies? for reify/record instances,
matching short names across qualified/bare protocol references.
- @x reads as the qualified clojure.core/deref, so it still derefs where a ns
excludes and rebinds deref (malli does). Updated reader-test + the reader
spec/grammar (S11, deref rule).
- Java collection interop on jolt collections: .nth/.count/.valAt/.get/.seq/
.containsKey route to the clojure.core equivalent (1-arg and 0-arg paths).
- java.util.HashMap capacity/load-factor constructors + .putAll.
- A class used as a value resolves to its instances' type, so Pattern -> the
regex type (malli keys class-schemas by it).
- Shims for malli's load path: LazilyPersistentVector/createOwning and
PersistentArrayMap/createWithCheck statics.
m/explain not yet working (jolt-fjb1). Full gate green.
* satisfies? recognizes reify, consistent with instance?
A reify's protocol methods are instance-local, so they aren't in the global type
registry that type-satisfies? consults — satisfies? returned false for a reify
even when it implemented the protocol. Check the protocols the reify records on
itself (the same :jolt/protocols list instance? uses), matching short names like
instance? does. Covers single- and multi-protocol reify.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add architecture refactor plan
Synthesizes a six-part architectural review into phased, gate-validated cleanup
work. Targets LLM-maintainability: one home per feature, no god-files, explicit
checked contracts, no copy-paste dispatch. No code changes yet — the plan only.
* Refactor phase 0: dead code + isolated bugs
Pure cleanup ahead of the structural phases (docs/architecture-refactor-plan.md).
No behavior change except the two bug fixes, which are covered by a regression row.
Dead code (all verified zero-reference or overridden):
- core-resolve / core-satisfies? / core-type->str seed stubs + bindings —
resolve and satisfies? are interned by install-stateful-fns! (the seed copies
were shadowed); type->str was an inert SCI stub with no callers.
- find defined twice in 20-coll.clj; the dead copy returned a plain vector
(wrong — the live def at :787 returns a real map-entry) with a comment that
contradicted it.
- mark-hint (passes.clj), phs-to-struct (phm), shape-vals / ns-imports-fn
(types) — unreferenced.
- redundant local pad2 in javatime (module-level one already in scope).
Bugs:
- File.toURL stored :url but every :jolt/url method reads :spec, so a URL from
(.toURL file) returned nil from all its methods. Now stores :spec (+ spec row).
- pl-rest had a no-op (if (plist? r) r r); collapsed to r.
- :map-shapes? was missing from the deps-image cache key — two runs differing
only in map-shapes could reuse each other's image.
Also dropped read-quote's unused pos param. Full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
A ^RecordType hint only resolved against the current namespace's ctor key, so a
hint naming a record defined in another namespace degraded to :any. That made a
decomposed multi-namespace program much slower than the monolith: per-namespace
inference can't see a record param's callers in other namespaces, and the
declared hint that could have typed it was dropped.
Resolution now works cross-namespace, for both record FIELD hints (defrecord)
and fn PARAM hints, in both spellings — ^Vec3 where the type is referred and
^v/Vec3 where the namespace is aliased:
- reader keeps a tag's namespace qualifier (^t/Ray -> "t/Ray", was "Ray").
- make-deftype-ctor-impl indexes each ctor closure by value; record-hint-ctor-key
resolves a hint name against the COMPILE ns (referred names live there; aliases
resolve through it) and maps the type var's root back to its home ctor key.
Using the ctor value, not the var's :ns, is what makes :refer work — :refer
re-interns a fresh var whose :ns is the referring ns.
- the analyzer captures record param hints as arity :phints [name ctor-key];
reinfer-def seeds those param types, so a record param is typed even with no
inferred caller — the open-world / cross-ns case.
Effect on the multi-namespace ray tracer: per-ns compile 30.4s -> 7.9s with
param hints, matching whole-program (8.1s) and the single-ns monolith (8.3s).
cross-ns-hints-test covers field + param hints, refer + as, and the reader tag.
A record field can carry a type hint — ^Vec3 (a defined record type) or ^:num —
and the inference now resolves it so reading the field back yields that exact type
instead of :any. A Vec3 stored in a Ray field reads out as Vec3, so the vec ops on
field-read values prove their reads (bare-index). This is Stalin's per-slot type
sets, but DECLARED rather than inferred: the exact shape is known up front.
- deftype captures each field's :tag / :num metadata (was stripped) and passes it
to make-deftype-ctor; the ctor registers per-field tags, resolving a record-type
hint to its ctor-key (same-ns) so the inference can look it up directly.
- call-ret-type builds a record's struct type with field types resolved from the
hints, recursing into nested record types (depth-bounded for self/cyclic types).
Measured: a nested-record read loop (:r (:origin ray)) runs 1.3s with ^Vec3 hints
vs 7.1s without — 5.5x. This is the lever the ray tracer needed (vecs flow through
container fields); records without it read back as :any and stay unproven.
A protocol method call compiles to (protocol-dispatch proto method this rest) — a
runtime registry walk (type-tag -> proto -> method) on every call, ~19x a direct
call. When the inference proves the receiver (arg 0) is a known record type, the
call now resolves to a DIRECT method call at compile time, skipping the registry.
- defprotocol registers each method's var-key 'ns/method' -> [proto method] (a
ctx-capturing register-protocol-methods! emitted into the do-block); infer-unit!
feeds it to the inference via a box (like record-shapes).
- the record-ctor return type carries :type (the record tag) so the inference
knows the receiver type; the :else invoke case annotates a protocol call whose
arg0 has a known :type with :devirt-{type,proto,method}.
- emit-invoke resolves the impl via find-protocol-method at emit time and emits a
direct call to the embedded impl fn value. Unknown/polymorphic receivers (no
proven :type) fall back to the dispatch path unchanged.
Measured: removes the dispatch overhead (14.7s -> 9.3s on a 10M-call loop); the
remaining cost is the method body itself (non-inlined, unproven reads) — inlining
the resolved method is the follow-up (jolt-t6r) toward direct-call speed.
Sound under the closed-world assumption direct-linking already makes (the impl is
resolved + embedded at compile time). Adds devirt-test (subprocess: dispatched ==
devirtualized across polymorphic dispatch, unknown-receiver fallback, and
heterogeneous collections). Stalin's compile-call/callee-environment is the model.
Records (defrecord/deftype) are now shape-recs in a direct-linking unit by
default — no JOLT_SHAPE flag. A record's shape is DECLARED, so the inference
proves field reads by a lookup, not fragile shape inference, and they bare-index.
Result: ~1.4x faster than the :jolt/deftype table form on a record-heavy loop
(3.9s vs 5.5s), driven by cheaper construction + proven bare-index reads.
Two gates now:
- :shapes? — shape-recs active; records use declared-shape layout + bare
index reads. On with direct-linking (where the inference runs).
- :map-shapes? — also shape generic const-key maps. Opt-in (JOLT_SHAPE), because
shaping maps net-loses on unproven reads (measured). Records win.
- call-ret-type types a record ctor (->Name) as a struct of its declared shape,
fed from a ctx-env registry populated at deftype; field reads on the result
bare-index. (set-record-shapes!/set-map-shapes! wired through infer-unit!.)
- sidx reads the field's position from the :shape vector AS-IS (declared order
for records, str-sorted for map literals) — no re-sort — so any field order
bare-indexes correctly. The map :map case only sets :shape under :map-shapes?.
- record-shape-for interns the descriptor per (type, fields), not per type: a
record redefined with different fields now gets a fresh descriptor instead of a
stale one (fixes redef descriptor staleness; old instances stay valid).
Adds record-declared-shape-test (declared-order reads, incl. non-alphabetical
fields, through fn boundaries + protocol method bodies). Known pre-existing edge
case filed as jolt-wf4 (direct (:f (->R …)) read returns nil after a record is
redefined with different fields; let-bound read works; repros without shapes).
The inference dropped the complete :shape whenever it rebuilt a struct type
(cap) or joined two (join-t/merge-fields), so a vec3 retrieved from a container
or a fn param typed across call sites lost its layout and every field read fell
to the slow descriptor path. Two fixes:
- cap preserves :shape: capping truncates field VALUES below the depth limit but
never the key SET, so the layout is still complete. It also recurses into
fields, so a shaped value nested in a container (a vec3 inside a hit-info)
keeps its own :shape — which is what lets (:r (:normal hit-info)) bare-index.
- join-t preserves :shape when both sides are the SAME complete shape (the
merged struct has the same keys); different shapes still drop it. This carries
the shape through if-joins and the inter-procedural fixpoint's call-site joins.
Result: the ray tracer goes from 22s (R1, correct-but-descriptor-path) to 4.36s
— 2.7x FASTER than the 11.7s no-shape baseline, and ~3x the JVM (was 8.5x), with
byte-identical output. The compounding of cheaper tuple construction plus
bare-index reads across the whole render far exceeds the per-op estimate.
Gate green flag-off, suite 4718, default-path bench even, transparency intact.
Removes the {:r :g :b} hardcoding. ANY constant key set is now a shape:
- inference: a struct type from a map LITERAL carries :shape (its canonical
str-sorted key vector — completeness); joins/access-inferred structs lack
it, so they never get a bare index. The literal node and lookup subjects
carry the shape; the back end derives the index from it.
- backend: emit-map turns any shape-tagged const-key map into a shape tuple;
emit-kw-lookup reads the field by bare index when the complete shape is
proven, else by the value's own descriptor (so a shape-rec whose :shape was
dropped by a join still reads correctly).
- runtime: core-get and core-assoc handle shape-recs.
Status: CORRECT for direct field access, container round-trips, and assoc
(minimal repros pass). NOT yet complete — the full ray tracer still hits an
uncovered path (a shape-rec reaching a map op without coverage: keys/vals/
count/seq/equality/print/jolt-call/dissoc/contains?/the interpreter's
coll-lookup all still need shape-rec branches). And the perf win needs
COMPLETENESS PRESERVATION through joins/containers (merge-fields/cap drop
:shape today, so nested vec3 access falls to the descriptor path, slower than
a struct get) — without it the general version is slower than the vec3
prototype.
All behind JOLT_SHAPE (off by default). Gate green with the flag off, suite
4718. This preserves the general design; the transparency layer + completeness
preservation are the remaining multi-session work.
Validated prototype of the hidden-class object-model change. A vec3-shaped
{:r :g :b} map literal is represented as a cheap Janet tuple [shape vb vg vr]
instead of a struct (~2x cheaper to construct); a lookup on a value the
inference PROVES is the shape reads by bare index with no runtime check.
Result on the ray tracer (direct-link): 12.3s -> 10.7s (~13% faster), with
byte-identical pixel output. The shape value flows transparently through
hit-info/ray/material containers and the colors vector; core-get handles it
(inline check, no fn call) so an unspecialized access is still correct.
Key lessons baked in: the lookup MUST compile to a bare index (a runtime
shape check, even inlined, taxed every field read and made it 2.5-3.4x
SLOWER) — so the inference gained a :shape hint (struct type with keys
exactly {:r :g :b}) that the back end turns into (in m idx). The descriptor
is quoted when embedded (its keys are a parens tuple Janet would otherwise
try to CALL).
All behind JOLT_SHAPE (off by default). Gate green, suite 4718, default-path
bench even. Scoped to the one shape; NOT yet sound in general (assumes every
vec3-shaped value is a shape-rec, true under the flag for the ray tracer) nor
fully transparent (only core-get + the inlined lookup; jolt-call/equality/
print/keys not yet covered). Those are the next steps toward a real feature.
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
File API (jolt-hjw): io/file and (File. …) build a tagged :jolt/file value
(instance? File true) with a full method surface (isFile/isDirectory/exists/
getName/getPath/getAbsolutePath/listFiles/toPath/delete/createNewFile/…) backed
by os/ and file/. file-seq is File-aware (leaves are File values). str/slurp/spit
coerce :jolt/file to its path. ClassLoader/getSystemClassLoader + a classloader
stub whose getResource returns nil degrade migratus's classpath lookup to the
filesystem. java.nio.file Path/FileSystem/PathMatcher are shimmed just enough for
script-excluded?'s glob (recursive * / ? matcher).
Three bugs found getting migratus's migration discovery to work:
- (assoc nil k v) returned a raw janet table, not a map, so assoc-in built tables
that count/seq rejected. Now returns an immutable map.
- methods/get-method resolved the multimethod symbol at runtime in the current
ns, so a bare multifn ref in its defining ns saw an empty table once defmethods
lived elsewhere. Now they take the multimethod VALUE and recover the var via a
registry (Clojure semantics).
- defmulti now drops a leading docstring/attr-map (migratus's multimethods carry
docstrings) instead of treating the docstring as the dispatch fn.
Conformance 335/335 x3, clojure-test-suite at baseline.
The analyzer always took (nth items 2) as the value, so (def x "doc" 42)
bound x to the docstring and dropped 42. Now it mirrors the interpreter:
when there are 4+ items and item 2 is a string, item 2 is the docstring
(attached as :doc meta) and item 3 is the value. Conformance 335/335 x3.
For the migratus next.jdbc shim (jolt-0z5):
- core.janet: __jdbc-wrap-conn / __jdbc-conn-raw / __jdbc-make-stmt builtins.
A connection is a tagged wrapper over a jdbc.core conn carrying a clj :exec
callback so the host Statement.executeBatch runs SQL without a janet->clj call.
- javatime.janet: tagged-methods for :jolt/jdbc-conn (setAutoCommit/isClosed/
close/getMetaData), :jolt/jdbc-meta (getDatabaseProductName), :jolt/jdbc-stmt
(addBatch/executeBatch/close); java.sql.Timestamp ctor -> millis.
- evaluator.janet: instance? case for Connection/java.sql.Connection so
migratus's do-commands runs SQL through its Connection branch.
Two defn/defmacro fixes found loading migratus.core (both rooted in the reader
representing ^{:map} name metadata as a with-meta form, jolt-8w2):
- defmacro special form: unwrap a with-meta name (mirrors def), and handle the
arity-clause form (defmacro name ([params] body...)) like fn/defn — a params
vector reads as a tuple, an arity clause as a list (array).
- defn overlay: pass the bare (unwrapped) name to fn while def keeps the meta.
Conformance 335/335 x3 modes.
Two provably-wrong cases the inference already has the facts for, closing the
last RFC 0006 open question:
- Calling a non-function. At an :invoke whose callee is provably :num or :str
(the only non-callable types — keywords/maps/vectors/sets are IFn), report
"cannot call a number as a function". Default level (no closed-world: the
callee type is inferred at the call site). Covers (5 1), ("hi" 0),
((+ 1 2) :k), a let-bound number, and a var holding a number (via vtype-box
in direct-link). A union is non-callable only when every member is, so
((if c 1 :k) x) is accepted (:kw is callable). Verified zero false positives
on the ray tracer, which calls maps/keywords/vectors as fns throughout.
- Wrong arity to a user fn. The registered single-fixed-arity sig (jolt-zo1)
makes a mismatched arg count provably throw; reported under the
JOLT_TYPE_CHECK_USER opt-in (same closed-world boundary; ^:redef/variadic
skipped). Caught at compile time before the runtime arity error.
Both fold into the existing infer walk, carry :pos for file:line:col, and keep
no-false-positives. Gate green, suite 4718, conformance 335/335, runtime bench
even (compile-time only).
Checking inherently needs an inference pass (~2.6x compile as a standalone
pass). But direct-link builds ALREADY run one inference pass for
specialization (run-passes' infer-top), so checking can ride along: set a
check-mode flag, turn checking? on during that existing pass, and collect
the diagnostics after — ~2% overhead measured on the ray tracer, vs 2.6x
for the separate pass.
So the checker now defaults to `warn` in direct-link builds (where it's
nearly free) and stays OFF in plain REPL/dev builds (no inference to ride,
no forced cost — opt in with JOLT_TYPE_CHECK there). JOLT_TYPE_CHECK still
overrides in both directions (off to disable, error to escalate).
It checks the POST-optimization IR, which matches what the optimized
program actually evaluates — scalar-replace only drops provably-pure code,
an accepted opt-mode divergence, so no real error is hidden. The loaders
enable position tracking whenever checking will run (env-selected or
direct-link). type-check! (the standalone pass) stays for plain builds;
both paths share report-diags!.
cli-test pins: plain build silent, direct-link warns by default,
JOLT_TYPE_CHECK=off disables. Gate green, suite 4718, runtime bench even.
The checker ran a separate check-walk that re-inferred each argument's
subtree AND recursed into it — quadratic in expression nesting. Fold the
diagnostic emission into `infer` itself (gated by a checking? flag, off
during the optimization fixpoint): one O(n) walk that both types and
checks. Removes check-walk entirely; check-form now drives infer.
This is a cleanup and removes the deep-nesting blowup, but it does NOT make
warn-by-default cheap: on a real 360-line file the checker still adds ~2.6x
compile time (277ms -> 720ms). That cost is the structural inference pass
itself, which checking inherently requires — not redundancy. A cheap
default-on path would need either piggybacking on the inference direct-link
already runs, or a lighter scalar-only checker inference. Gate green,
type-check tests pass.
RFC 0006 error reporting wanted file:line:col but IR nodes carried no
position, so diagnostics read only "type error in <ns>: <msg>". Now:
type error /tmp/scene.clj:5:5: `inc` requires a number, but argument 1 is a string
The reader records each LIST form's absolute start offset in a table keyed
by form identity (lists are fresh arrays, never interned), gated behind a
flag the loaders enable only when JOLT_TYPE_CHECK is on — zero cost off.
Keying by identity makes positions survive macroexpansion exactly when the
user's own sub-form is spliced through, and absent for macro-synthesized
structure: a `(inc :k)` written inside `(when c ...)` reports at its own
line, never at the expansion's generated if/do.
The analyzer stamps the offset onto :invoke nodes (form-position host
contract fn); the checker carries it into each diagnostic as :pos; the
loaders stash the file's source + path on the env (save/restored across
nested requires); backend/type-check! converts offset -> line:col via the
reader's line-col and renders the RFC format. Falls back to the ns when no
position is available (synthetic forms), so it is never worse than before.
Gate green, conformance 335/335, suite 4718, runtime bench even (positions
are compile-time only; off by default).
The success checker fired only against core-fn error domains (stable, not
redefinable). This adds reporting of a call that passes a provably-wrong
type to a USER fn whose body requires otherwise — e.g. a fn that only does
arithmetic on a param, called with a string.
As check-walk sees defs it registers each non-redefinable single-fixed-arity
user fn's {:params :body} in module state (user-sig-box, accumulating across
forms like rtenv-box — a def must precede its call). At a call site (strict
mode only) the body is re-checked with ONE parameter bound to its concrete
argument type, others :any; if that produces a diagnostic the all-:any body
did not, the argument alone is provably wrong and the call is reported.
Monotonic — binding a concrete type can only add error-domain hits — so still
no false positives. A cycle guard (checking-box) terminates mutual recursion.
Gated behind JOLT_TYPE_CHECK_USER (orthogonal to the warn/error level)
because it rests on the closed-world assumption, weaker than the core-fn
case. check-form gains a strict? arity; the default path is unchanged and
user-fn code runs only when the checker is enabled. ^:redef/^:dynamic and
multi/variadic fns are not registered (their body is no stable requirement).
Gate green, suite 4718, conformance 335/335.
The success checker (RFC 0006) used to lose differing if-branches to :any
and accept the use. (inc (if c "a" :k)) typed the if as :any — sound but
imprecise, since the value is provably {:str | :kw}, every member of which
is in inc's error domain.
Adds {:union #{T...}} to the lattice: join-t forms a scalar union of
differing branches instead of collapsing to :any, capped at 4 distinct
scalars (the member space is the five scalar tags, so the lattice stays
finite and the inter-procedural fixpoint still terminates). The checker's
not-number?/not-seqable? report a union only when EVERY member is in the
error domain — any valid member accepts the call, so still no false
positives. type-name renders "a string or a keyword".
Unions are scalar-only and carry no :struct/:vec/:set key, so every
structural predicate already treats them as opaque — specialization sees
them exactly as :any and codegen is unchanged. Gate green, suite 4718,
conformance 335/335, bench even.
Reuse the structural inference from RFC 0005 as a loose type checker. It reports
a core-fn call only when an argument's inferred type is concrete and lies in
that op's throwing error domain, and accepts everything ambiguous (:any, a
union that joined to :any, :truthy). By construction it never produces a false
positive: a correct program has nothing to report even in error mode.
The curated error-domain table starts with the clearest throwing cases:
arithmetic on a provable non-number, and count/first/rest/next/seq/nth on a
provable non-seqable scalar. Lenient operations like (get 5 :k) and (:k 5),
which return nil rather than throw, are deliberately not listed.
Checking is decoupled from specialization: it runs whenever JOLT_TYPE_CHECK is
warn or error, regardless of :inline?, reading the knob at compile time so no
rebuild is needed. warn prints to stderr, error fails the form's compilation,
off (the default) skips it entirely. Core init stays clean under the flag.
jolt-y3b
Replace the ad-hoc inference lattice (a flat :struct-map tag plus {:vec ELEM})
with one recursive structural type: {:struct {field -> T}}, {:vec T}, {:set T},
scalar tags, and :any. A keyword lookup now returns its field's type, so nested
access like (:r (:direction ray)) is typed end to end and drops its guard. join
is field-wise and element-wise with a depth cap of 4 so the inter-procedural
fixpoint still terminates.
The back end honors a struct hint on any subject node, not just locals, so an
inferred field type on a nested lookup specializes. The orchestrator's fixpoint
joins through the portable join-types so compound types no longer collapse to
:any.
Ray tracer goes 12.8s to 11.0s with no hints, matching the explicit ^:struct
version (10.9s). Render checksum unchanged (1915337), full gate green,
conformance x3 modes pass.
jolt-5uj
The inference now tags a :local it proved to be a vector with :hint :vector, and
the back end specializes (count v) -> pv-count (skipping core-count's dispatch
chain) and the 3-arg (nth v i default) -> pv-nth. The 2-arg nth is deliberately
NOT specialized: pv-nth returns nil out-of-bounds where Clojure nth throws.
Sound, conformance 335/335 x3 and full jpm test pass; type-infer-phase2-test
pins the specialization and the 2-arg exclusion.
Extends the inference lattice with a parametric vector type {:vec ELEM} and
threads element types through the program:
- vector literals, conj/into, and range produce element-typed vectors;
- reduce/map/mapv/filter/filterv seed their closure's element (and reduce's
accumulator) param, so a lookup inside the closure over a vector-of-structs
specializes (the HOF-element-awareness piece);
- a var reference carries a VALUE type — a fn var is :truthy (non-nil, sealed
root), a def var carries its inferred init type (e.g. a color table is
{:vec :struct-map}); element-returning fns (rand-nth/first/nth/...) yield the
collection's element type. These let the dynamically-built scene's sphere
maps type as structs.
The inter-procedural fixpoint now also infers non-fn def value types, and the
recompile re-emits the WHOLE unit callee-first (reverse-topological) so a
caller re-embeds its recompiled, now-specialized callees and a call site
compiled after the pass links the whole chain.
Result on the ray tracer (no hints): the chain closes — hittables infers to
{:vec :struct-map}, hit-sphere's hittable param to :struct-map — and the render
goes 13.1s -> 12.8s. That is only ~3%, far short of the explicit hint's 1.22x.
The remaining gap is nested field access: a lookup RESULT like (:direction ray)
is :any, so (:r (:direction ray)) stays guarded, and the vec3 fns (called with
such values) can't be typed struct. The hint asserts the vec3 params directly
and propagates through inlining; matching it needs field-shape types
(ray.direction : vec3, vec3.r : number) — a structural extension (Phase 4).
Sound: a seeded full render produces an identical checksum (1915337);
conformance 335/335 x3 and the full jpm test pass; type-infer-phase3-test pins
the element-typing + HOF mechanism. Phase 2 (vector nth/count specialization)
was deprioritized — it is orthogonal to this benchmark.
Closed-world (optimization mode): after a unit loads, infer-unit! runs a
whole-unit fixpoint over the call graph and recompiles. A fn's param types are
the lub of its in-unit call-site arg types; its return type is the lub of its
tail positions; iterated to a least fixpoint. Param types are RECOMPUTED FRESH
each iteration (not accumulated) because :any is the lattice top — joining an
early-iteration :any would poison the result permanently. Closures inherit the
enclosing tenv so captured locals keep their types (their own params shadow to
:any). A fn whose var escapes as a VALUE keeps :any params (its callers aren't
all visible). Each fn is then re-inferred with its param types seeded and
re-emitted; recompiled bodies are semantically identical, so correctness holds
regardless of order. Sound under source distribution + whole-program compile
(the consumer compiles all call sites together).
Plumbing: the portable pass (jolt.passes) gained inter-procedural primitives —
set-rtenv!, infer-body (types a body, collects its call sites), reinfer-def
(seeds param types), and escape tracking. The back end stashes each
single-fixed-arity defn's :def IR (:infer-ir); the evaluator triggers
infer-unit! after a unit loads (via an env hook, opt mode only).
Result and honest finding: the fixpoint correctly types scalar-flowing params
(ray-cast/hit-all/hit-sphere all get the ray param as :struct-map, no hint),
but the ray tracer does NOT speed up — its dominant lookups are on `hittable`,
the element of the `hittables` vector threaded through `reduce`, which stays
:any. Typing it needs collection-element types (vector<struct>) plus HOF-element
awareness (knowing reduce applies the closure to elements), which is beyond
inter-procedural param inference. The explicit ^:struct hint reaches it (it
types the reduce closure param directly), which is why the hinted run is 1.22x.
Verified: conformance 335/335 x3, full jpm test; new type-infer-phase1-test
pins the fixpoint, the escape gate, the seeded re-inference, and correctness.
A forward, soft-typing-style pass (simplified HM: monovariant, never-fails,
lattice top = :any) in jolt.passes, run after the inline/scalar-replace
fixpoint when the optimization mode is on. It types expressions from literals
and arithmetic, flows the type through let bindings, and joins at if-branches.
Where a keyword-lookup subject is PROVEN to be a plain struct map it sets
:hint :struct (the same channel a manual hint uses, so the back end drops the
:jolt/type guard); where the type is :any it leaves the dynamic guard in place.
Sound by construction: a concrete type is assigned only when proven (scalar
keys with non-nil/non-false values for a struct-map), so a wrong bare get can't
happen. This is the foundation; on its own it mostly overlaps Route 1
scalar-replacement (which already eliminates non-escaping let-bound maps), so
its standalone win is small. Phase 1 (inter-procedural) is where escaping
params get typed.
Verified: conformance 335/335 x3, full jpm test; new type-infer-test pins the
flow rules and the sound :any fallback (cases force the map to escape so the
test isolates inference from scalar-replacement).
Builds on the ^:struct keyword-lookup hint:
- ^TypeName for records. A tag naming a defrecord/deftype now resolves to the
struct fast path: record instances are tables tagged :jolt/deftype (not
:jolt/type), so a raw keyword get is correct for them. A new host contract fn
record-type? detects a record by its ->Name constructor; a non-record tag
(^String, ^long, ...) is ignored, as before.
- (get m :k) and (get m :k default) now get the same inlined keyword lookup as
(:k m): the representation guard fast path when unhinted, and the bare get
when the subject is ^:struct/^Record. A variable/number/string key still
falls through to core-get. The two call shapes share one emitter
(emit-kw-lookup).
- JOLT_CHECK_HINTS=1 turns a violated hint into a clear runtime error (naming
the local and key) by keeping the guard and throwing on the tagged arm. It is
off by default with zero cost to normal builds (a hinted lookup still emits a
bare get), and is part of the image-cache fingerprint. This is the answer to
"a lying hint is silent": opt into checking during development.
- Docs: RFC 0004 records the design, soundness contract, and measurements; the
reader spec gains S12b (hints are semantically transparent; jolt recognizes
^:struct and ^Record as lookup-optimization assertions).
There is no Clojure keyword equivalent for "plain map / fast keyword access"
(Clojure hints are class names), so ^:struct stays a jolt-specific flag,
analogous to ^:dynamic.
Verified: conformance 335/335 in all three modes and the full jpm test pass; a
seeded ray-tracer render is byte-identical hinted vs unhinted; the struct-hint
test covers record hints, the get-form, inline propagation, and the checked-mode
error. Full render with hints holds at 13.3s -> 10.9s (1.22x).
A constant-keyword lookup (:k m) currently emits a guarded form,
(if (get m :jolt/type) (core-get m k) (get m k)), to tell a plain struct
(raw get is correct) from a phm/sorted/transient (needs core-get). On a
struct that guard is a second get, so the lookup costs ~36ns where a bare
get is ~20ns. Profiling the ray tracer (jolt-dad) showed keyword lookups are
~50% of a render and the guard is the only avoidable part, but dropping it
needs to know statically that the subject is a plain struct.
Type hints are exactly that information, and jolt already parses them and
otherwise ignores them. This wires one through: a local hinted ^:struct
asserts a plain struct/record map, so a (:k local) lookup on it skips the
guard and emits a bare get. The hint rides on the binding symbol into the
analyzer, which records it per-local and attaches it to :local IR nodes; the
back end reads it on the lookup subject. It also propagates through inlining:
when the inliner let-binds a non-trivial arg to a fresh local, it carries the
called fn's param hint onto that local, so lookups inside the spliced body
keep the bare path. This is a programmer assertion, like a Clojure type hint
(an inaccurate hint just makes the raw get return the wrong value, the same
contract as a wrong ^String), so it stays opt-in and off by default.
On the ray tracer (with inlining on) this is 13.3s to 10.9s, 1.22x, taking it
to 7.8x JVM from 9.4x after the inline pass. The unhinted path emits identical
code (the fast arm is just factored out), so nothing changes without hints.
Verified: a seeded full render produces an identical checksum hinted vs
unhinted; conformance 335/335 in all three modes and the full jpm test pass;
new test/integration/struct-hint-test.janet pins the guard removal, the
inline propagation, and that an accurate hint is correctness-preserving.
Adds two IR passes to jolt.passes that run when a unit opts into
direct-linking (JOLT_DIRECT_LINK=1, off by default). The inline pass splices
small direct-linked fns at their call sites, copy-propagating trivial args so
that scalar replacement can then see map literals across the call boundary.
Scalar replacement is AOT escape analysis: a map allocation whose only use is
constant-keyword lookup is dropped and each (:k m) is replaced with the value
at :k, both for a literal lookup subject and for a non-escaping let-bound map.
Inlining and scalar replacement iterate to a capped fixpoint, since inlining
exposes literals that scalar replacement then collapses.
The back end stashes the body IR of each single-fixed-arity defn on its var
cell (inline-stash!), and the portable pass reads it through two new jolt.host
contract fns (inline-enabled?, inline-ir). Inlining is gated on :inline?, which
is off for all of init so core and the self-hosted compiler compile exactly as
before (const-fold only); api/init and main re-read JOLT_DIRECT_LINK so the
flag works both for a freshly built context and for the build-time-baked one in
the shipped binary.
Only inline-safe targets are spliced: a single fixed arity, no recur/loop/fn/
try crossing the boundary, within a size budget, a closed body (no free locals
beyond the params, so a self-recursive fn's name reference can't dangle), and
not ^:redef / ^:dynamic. Bodies are fully alpha-renamed so no spliced name can
collide with a caller local.
On the ray tracer this is 15.3s -> 13.0s (1.18x). The ceiling is honest: that
workload's cost is dominated by lookups on maps that genuinely escape (rays,
hits, materials) and by dynamic dispatch (the reduce closure, the :scatter fn),
which escape analysis cannot remove. On allocation-bound code where the
temporaries are local it is far larger: a vec3 reflect+dot loop goes 9.3s ->
0.38s (25x), with the loop body reduced to pure arithmetic.
Verified: full jpm test passes (inline off, no regression); conformance 335/335
in all three modes and the clojure-test-suite both pass with inline on; new
inline-sra-test pins the transform and its semantics.
A typo'd symbol used to auto-intern an unbound var and die later as
'Cannot call nil as a function' with no hint which symbol. Now:
$ jolt -e '(undefined-fn 1)'
Error: Unable to resolve symbol: undefined-fn in this context
The analyzer's :unresolved fallthrough now punts to the interpreter
(whose resolver raises the message above when the form runs) instead of
emitting a var-ref that interned the var. A punt rather than a hard
throw because runtime-interning forms (defmulti's setup) legitimately
reference the var they're about to create from a nested do.
Pulling that thread surfaced three real bugs the leniency was masking:
- h-resolve-global resolved unqualified symbols against ctx-current-ns,
which during analysis is jolt.analyzer — so user-ns vars NEVER
resolved through it; the lenient arm happened to emit the right ns.
Now resolves against the compile ns like the qualified branch.
- Top-level (do ...) wasn't split: Clojure compiles and EVALS each
child in sequence so earlier children's runtime effects (defmulti's
intern) are visible while later children compile. eval-toplevel now
splits.
- The stdlib itself had forward references the auto-intern hid:
10-seq's transducers used vreset!/vswap! from 20-coll (moved to
10-seq); in 20-coll qualified-ident?/realized?/list*/underive
referenced defs declared later in the file (reordered); sorted? and
partition-all are genuinely later-tier and got (declare ...).
Test rows updated where they encoded the old leniency: ir-passes'
dead-branch row (unresolved in a dead branch is an error, as in
Clojure), compile-mode's ctx-isolation row (other ctx now errors
instead of reading nil), cli rows assert the new message. Gate green,
conformance 335/335 x3, suite 4718 steady, bench within noise.
Before: (+ 1 "a") printed 'could not find method :+ for 1 or :r+ for "a"'
followed by three janet frames pointing at jolt internals. After:
Error: Cannot add 1 and "a" — + expects numbers
at app.deep/level3
Round 1 — compiled fns carry their Clojure identity:
- The analyzer's recur target (which doubles as the compiled janet fn's
name) is now ns/fn-name (_r$app.deep/level3--N), so janet stack traces
name the user's fns; defn passes the self-name through to fn.
- eval-toplevel re-raises with propagate instead of protect+error — the
failing fiber's stack was being discarded, which is why every trace began
at eval-toplevel.
- require/maybe-require-ns route loaded namespaces through the loader's
compile-or-interpret eval-toplevel via a ctx hook (the evaluator can't
import the loader). Previously REQUIRED namespaces always ran interpreted:
slower, and their fns were anonymous in traces.
Round 2 — report-error presents for users (rephrase-inspired):
- The full trace text is stashed at the innermost eval-toplevel boundary
(janet's debug/stacktrace walks the fiber propagation chain; debug/stack
cannot), then filtered: _r$ frames demangled to ns/fn-name, jolt-internal
and [eval] frames dropped. JOLT_DEBUG=1 restores the raw janet trace.
- Message rewrites: janet arithmetic dispatch -> 'Cannot add X and Y — +
expects numbers'; compiled arity -> Clojure's 'Wrong number of args (N)
passed to: ns/fn'; nil-call gets an undefined-symbol hint (round 3 will
fix resolution properly).
6 cli-test rows assert the exact user-visible output. Gate green, suite
4718 steady, bench within noise.
