Records (defrecord/deftype) are now shape-recs in a direct-linking unit by
default — no JOLT_SHAPE flag. A record's shape is DECLARED, so the inference
proves field reads by a lookup, not fragile shape inference, and they bare-index.
Result: ~1.4x faster than the :jolt/deftype table form on a record-heavy loop
(3.9s vs 5.5s), driven by cheaper construction + proven bare-index reads.
Two gates now:
- :shapes? — shape-recs active; records use declared-shape layout + bare
index reads. On with direct-linking (where the inference runs).
- :map-shapes? — also shape generic const-key maps. Opt-in (JOLT_SHAPE), because
shaping maps net-loses on unproven reads (measured). Records win.
- call-ret-type types a record ctor (->Name) as a struct of its declared shape,
fed from a ctx-env registry populated at deftype; field reads on the result
bare-index. (set-record-shapes!/set-map-shapes! wired through infer-unit!.)
- sidx reads the field's position from the :shape vector AS-IS (declared order
for records, str-sorted for map literals) — no re-sort — so any field order
bare-indexes correctly. The map :map case only sets :shape under :map-shapes?.
- record-shape-for interns the descriptor per (type, fields), not per type: a
record redefined with different fields now gets a fresh descriptor instead of a
stale one (fixes redef descriptor staleness; old instances stay valid).
Adds record-declared-shape-test (declared-order reads, incl. non-alphabetical
fields, through fn boundaries + protocol method bodies). Known pre-existing edge
case filed as jolt-wf4 (direct (:f (->R …)) read returns nil after a record is
redefined with different fields; let-bound read works; repros without shapes).
Measurement (vec-heavy benchmark + read-mechanism micro-benchmarks) showed that
shape-rec'ing generic const-key maps is a net loss in a bytecode VM: an unproven
field read can't beat a native Janet struct-get (one opcode), and an inline cache
doesn't transfer to a VM jolt doesn't control. The win is real only for PROVEN
reads (bare-index beats struct) — i.e. construction-heavy or inference-friendly
code, not general map-through-fn code.
So shapes are opt-in again (JOLT_SHAPE), not defaulted on in direct-link. Kept the
get-or-shape fix: the non-proven shape read now indexes inline off the descriptor
instead of calling core-get (which was ~2.5x slower per read). :shapes? is the
single opt-in gate; image cache keys on JOLT_SHAPE + JOLT_NO_SHAPE.
Next: records with declared shapes get fast field access by default (the general
case), which is where the proven-read win actually lands.
Record shape-recs now compare type-aware like the table form: eq-map-pairs
returns nil for a record so equality falls to deep=, which is type-aware via the
per-type interned descriptor. A record equals only a same-type record, never a
plain map (cross-type and record-vs-map were wrongly equal under JOLT_SHAPE).
assoc of a new (undeclared) key keeps the record type and grows a slot, matching
Clojure's record-extension semantics.
Adds test/integration/record-shape-test.janet locking in the runtime record
mechanism (tag, field access, virtual :jolt/deftype, type-preserving assoc,
dissoc demotion, type-aware equality, #ns.Type{...} printing).
Records (defrecord/deftype) become shape-recs whose descriptor also carries
:type, under JOLT_SHAPE (flag off keeps the :jolt/deftype table form). A record
is a Janet tuple [descriptor field0 ...] in declared field order, so the
positional ->Name ctor maps args straight to slots.
- record-tag unifies the type accessor over both reps; instance?/satisfies?/
protocol dispatch and the . interop path go through it.
- virtual :jolt/deftype key on record shape-recs (via shape-get) keeps every
existing (get obj :jolt/deftype) dispatch site working.
- emit-kw-lookup's non-proven fallback routes any tuple to core-get (shape
aware), not gated on compile-time JOLT_SHAPE — core is baked without the flag
but still receives user shape-recs.
- assoc keeps the type (same-shape in place, new key grows a slot Clojure-style);
dissoc of a declared field demotes to a plain map; pr prints #ns.Type{...}.
- JOLT_SHAPE added to the image-cache key (it shapes core's own compiled paths).
The inference dropped the complete :shape whenever it rebuilt a struct type
(cap) or joined two (join-t/merge-fields), so a vec3 retrieved from a container
or a fn param typed across call sites lost its layout and every field read fell
to the slow descriptor path. Two fixes:
- cap preserves :shape: capping truncates field VALUES below the depth limit but
never the key SET, so the layout is still complete. It also recurses into
fields, so a shaped value nested in a container (a vec3 inside a hit-info)
keeps its own :shape — which is what lets (:r (:normal hit-info)) bare-index.
- join-t preserves :shape when both sides are the SAME complete shape (the
merged struct has the same keys); different shapes still drop it. This carries
the shape through if-joins and the inter-procedural fixpoint's call-site joins.
Result: the ray tracer goes from 22s (R1, correct-but-descriptor-path) to 4.36s
— 2.7x FASTER than the 11.7s no-shape baseline, and ~3x the JVM (was 8.5x), with
byte-identical output. The compounding of cheaper tuple construction plus
bare-index reads across the whole render far exceeds the per-op estimate.
Gate green flag-off, suite 4718, default-path bench even, transparency intact.
Removes ALL hardcoding. Every constant-key map literal in user code becomes a
shape-rec (a Janet tuple [descriptor v0 v1 ...]); the descriptor is interned
per key SET with a single canonical key order owned by the runtime
(types/shape-sort), so every site that builds or reads a shape agrees.
Consistency is the foundation: shape-recs are made UNCONDITIONALLY for
const-key user maps (emit-map), not gated on the inference — so a value's
representation always matches what the type system claims, which was the bug
that broke the earlier generalization (a ray built as a struct but bare-indexed
as a shape). Gated on :inline? so it applies to user data only, never to core
or the compiler's own IR-node maps.
Transparency layer (the shape-rec block now lives in types.janet, reachable by
core + evaluator + backend): get, assoc, dissoc, count, contains?, map?, first,
seq, the central realize-for-iteration normalizer (keys/vals/reduce-kv), eq-map-
pairs + eq-seqable (equality), pr-render (print), jolt-call (compiled IFn), and
the interpreter's coll-lookup all handle shape-recs. A new spec
(shape-transparency-test) asserts each op matches the equivalent struct map,
including nil/false values (which shape-recs store positionally — unlike
structs).
The ray tracer renders byte-identically (mean 122.04). Gate green with the flag
off, suite 4718. NOT yet faster — every field read currently takes the
descriptor path because the inference drops the complete :shape through joins
and containers; that's Round 2 (completeness preservation). All behind
JOLT_SHAPE (off by default).
Removes the {:r :g :b} hardcoding. ANY constant key set is now a shape:
- inference: a struct type from a map LITERAL carries :shape (its canonical
str-sorted key vector — completeness); joins/access-inferred structs lack
it, so they never get a bare index. The literal node and lookup subjects
carry the shape; the back end derives the index from it.
- backend: emit-map turns any shape-tagged const-key map into a shape tuple;
emit-kw-lookup reads the field by bare index when the complete shape is
proven, else by the value's own descriptor (so a shape-rec whose :shape was
dropped by a join still reads correctly).
- runtime: core-get and core-assoc handle shape-recs.
Status: CORRECT for direct field access, container round-trips, and assoc
(minimal repros pass). NOT yet complete — the full ray tracer still hits an
uncovered path (a shape-rec reaching a map op without coverage: keys/vals/
count/seq/equality/print/jolt-call/dissoc/contains?/the interpreter's
coll-lookup all still need shape-rec branches). And the perf win needs
COMPLETENESS PRESERVATION through joins/containers (merge-fields/cap drop
:shape today, so nested vec3 access falls to the descriptor path, slower than
a struct get) — without it the general version is slower than the vec3
prototype.
All behind JOLT_SHAPE (off by default). Gate green with the flag off, suite
4718. This preserves the general design; the transparency layer + completeness
preservation are the remaining multi-session work.
Validated prototype of the hidden-class object-model change. A vec3-shaped
{:r :g :b} map literal is represented as a cheap Janet tuple [shape vb vg vr]
instead of a struct (~2x cheaper to construct); a lookup on a value the
inference PROVES is the shape reads by bare index with no runtime check.
Result on the ray tracer (direct-link): 12.3s -> 10.7s (~13% faster), with
byte-identical pixel output. The shape value flows transparently through
hit-info/ray/material containers and the colors vector; core-get handles it
(inline check, no fn call) so an unspecialized access is still correct.
Key lessons baked in: the lookup MUST compile to a bare index (a runtime
shape check, even inlined, taxed every field read and made it 2.5-3.4x
SLOWER) — so the inference gained a :shape hint (struct type with keys
exactly {:r :g :b}) that the back end turns into (in m idx). The descriptor
is quoted when embedded (its keys are a parens tuple Janet would otherwise
try to CALL).
All behind JOLT_SHAPE (off by default). Gate green, suite 4718, default-path
bench even. Scoped to the one shape; NOT yet sound in general (assumes every
vec3-shaped value is a shape-rec, true under the flag for the ray tracer) nor
fully transparent (only core-get + the inlined lookup; jolt-call/equality/
print/keys not yet covered). Those are the next steps toward a real feature.
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
Two try bugs that blocked migratus's migrate (run/table-exists? are multi-body
try/catch/finally with janet-interop jdbc calls, which punt to the interpreter):
- The interpreter's try took only form 1 as the body, dropping later body forms
before the clauses, and did not run finally on the success path when a catch
was present. Rewritten to collect every body form before the first
catch/finally and to always run finally (via defer, so it fires even if a
catch body throws).
- current-ns leaked after a caught throw from an INTERPRETED fn (it sets ns to
its defining ns and can't restore on unwind). The interpreter's try now
restores; the compiled try (emit-try) snapshots the ns at entry and resets it
in the catch via new __current-ns/__set-current-ns! core fns. Statement
values also get a :close key so with-open can close them.
With these, migratus migrate/rollback round-trips on jolt (verified end to end
against sqlite). Conformance 335/335 x3, full gate green.
methods/get-method now take the multimethod VALUE (Clojure semantics), so the
arg must resolve to compile. The isolated analyzer never defines mf, so point
these two at print-method (a real defmulti) instead.
File API (jolt-hjw): io/file and (File. …) build a tagged :jolt/file value
(instance? File true) with a full method surface (isFile/isDirectory/exists/
getName/getPath/getAbsolutePath/listFiles/toPath/delete/createNewFile/…) backed
by os/ and file/. file-seq is File-aware (leaves are File values). str/slurp/spit
coerce :jolt/file to its path. ClassLoader/getSystemClassLoader + a classloader
stub whose getResource returns nil degrade migratus's classpath lookup to the
filesystem. java.nio.file Path/FileSystem/PathMatcher are shimmed just enough for
script-excluded?'s glob (recursive * / ? matcher).
Three bugs found getting migratus's migration discovery to work:
- (assoc nil k v) returned a raw janet table, not a map, so assoc-in built tables
that count/seq rejected. Now returns an immutable map.
- methods/get-method resolved the multimethod symbol at runtime in the current
ns, so a bare multifn ref in its defining ns saw an empty table once defmethods
lived elsewhere. Now they take the multimethod VALUE and recover the var via a
registry (Clojure semantics).
- defmulti now drops a leading docstring/attr-map (migratus's multimethods carry
docstrings) instead of treating the docstring as the dispatch fn.
Conformance 335/335 x3, clojure-test-suite at baseline.
The analyzer always took (nth items 2) as the value, so (def x "doc" 42)
bound x to the docstring and dropped 42. Now it mirrors the interpreter:
when there are 4+ items and item 2 is a string, item 2 is the docstring
(attached as :doc meta) and item 3 is the value. Conformance 335/335 x3.
For the migratus next.jdbc shim (jolt-0z5):
- core.janet: __jdbc-wrap-conn / __jdbc-conn-raw / __jdbc-make-stmt builtins.
A connection is a tagged wrapper over a jdbc.core conn carrying a clj :exec
callback so the host Statement.executeBatch runs SQL without a janet->clj call.
- javatime.janet: tagged-methods for :jolt/jdbc-conn (setAutoCommit/isClosed/
close/getMetaData), :jolt/jdbc-meta (getDatabaseProductName), :jolt/jdbc-stmt
(addBatch/executeBatch/close); java.sql.Timestamp ctor -> millis.
- evaluator.janet: instance? case for Connection/java.sql.Connection so
migratus's do-commands runs SQL through its Connection branch.
Two defn/defmacro fixes found loading migratus.core (both rooted in the reader
representing ^{:map} name metadata as a with-meta form, jolt-8w2):
- defmacro special form: unwrap a with-meta name (mirrors def), and handle the
arity-clause form (defmacro name ([params] body...)) like fn/defn — a params
vector reads as a tuple, an arity clause as a list (array).
- defn overlay: pass the bare (unwrapped) name to fn while def keeps the meta.
Conformance 335/335 x3 modes.
Faithful shim of the real clojure/tools.logging public API rather than a
bespoke logger. clojure.tools.logging.impl provides the Logger/LoggerFactory
protocols (matching upstream signatures) plus a jolt stderr-backed factory;
find-factory returns it instead of probing slf4j/log4j/jul. The macro surface
(logp/logf, level macros + f-variants, log, enabled?, spy) dispatches through
those protocols. Departures forced by no-JVM: log* writes directly (no agent/
LockingTransaction), and the throwable-first-arg branch is dropped since
(instance? Throwable x) is always false for jolt exception values.
Workarounds for two jolt gaps found en route (filed as jolt-9av, jolt-6ym):
macro bodies fully-qualify impl refs because syntax-quote does not resolve ns
aliases, and def docstrings are kept in comments because (def name doc value)
is mishandled.
stderr via __eprint. trace/debug suppressed by default (impl/*level* :info).
Conformance 335/335 x3 modes.
Two provably-wrong cases the inference already has the facts for, closing the
last RFC 0006 open question:
- Calling a non-function. At an :invoke whose callee is provably :num or :str
(the only non-callable types — keywords/maps/vectors/sets are IFn), report
"cannot call a number as a function". Default level (no closed-world: the
callee type is inferred at the call site). Covers (5 1), ("hi" 0),
((+ 1 2) :k), a let-bound number, and a var holding a number (via vtype-box
in direct-link). A union is non-callable only when every member is, so
((if c 1 :k) x) is accepted (:kw is callable). Verified zero false positives
on the ray tracer, which calls maps/keywords/vectors as fns throughout.
- Wrong arity to a user fn. The registered single-fixed-arity sig (jolt-zo1)
makes a mismatched arg count provably throw; reported under the
JOLT_TYPE_CHECK_USER opt-in (same closed-world boundary; ^:redef/variadic
skipped). Caught at compile time before the runtime arity error.
Both fold into the existing infer walk, carry :pos for file:line:col, and keep
no-false-positives. Gate green, suite 4718, conformance 335/335, runtime bench
even (compile-time only).
Character/isUpperCase + isLowerCase (ASCII, on :jolt/char structs);
Thread/interrupted (false, no real threads) + Thread/currentThread stub
with a .getContextClassLoader classloader stub; Long/valueOf; java.net.URI
constructor (string round-trip); and a minimal java.util.Date / TimeZone /
SimpleDateFormat supporting the y M d H m s tokens migratus's timestamp uses,
backed by os/date (UTC default). Full gate + 3-mode conformance green.
jolt-6xk: resolve bare exception class symbols (Exception,
IllegalArgumentException, InterruptedException, Throwable) by consulting
class-ctors/class-canonical-names in the unqualified symbol-resolution
fallthrough, mirroring the qualified path. Constructors already existed
in javatime; throw/catch/.getMessage already handled the string payload.
jolt-47b: java.util.regex.Pattern statics (compile/quote/MULTILINE) that
return jolt's native :jolt/regex value so str/replace, re-matches, and
.split accept them transparently. MULTILINE maps to a (?m) prefix routed
through the regex engine's inline-flag path; the engine gains (?m) anchor
support with the non-multiline branches left byte-identical. String
methods .matches/.replaceAll/.replaceFirst added. .split dispatches on
compiled-regex via a :jolt/regex tagged-method.
Full jpm test gate green.
Update the status, strictness levels, and open questions to reflect what
landed: bounded unions (jolt-pz5), user-fn domains behind
JOLT_TYPE_CHECK_USER (jolt-zo1), precise file:line:col (jolt-fqy), and the
checker folded into one inference walk that piggybacks on direct-link
specialization (on by default there, opt-in in plain builds). Align the
error-reporting example with the actual output format.
Checking inherently needs an inference pass (~2.6x compile as a standalone
pass). But direct-link builds ALREADY run one inference pass for
specialization (run-passes' infer-top), so checking can ride along: set a
check-mode flag, turn checking? on during that existing pass, and collect
the diagnostics after — ~2% overhead measured on the ray tracer, vs 2.6x
for the separate pass.
So the checker now defaults to `warn` in direct-link builds (where it's
nearly free) and stays OFF in plain REPL/dev builds (no inference to ride,
no forced cost — opt in with JOLT_TYPE_CHECK there). JOLT_TYPE_CHECK still
overrides in both directions (off to disable, error to escalate).
It checks the POST-optimization IR, which matches what the optimized
program actually evaluates — scalar-replace only drops provably-pure code,
an accepted opt-mode divergence, so no real error is hidden. The loaders
enable position tracking whenever checking will run (env-selected or
direct-link). type-check! (the standalone pass) stays for plain builds;
both paths share report-diags!.
cli-test pins: plain build silent, direct-link warns by default,
JOLT_TYPE_CHECK=off disables. Gate green, suite 4718, runtime bench even.
The checker ran a separate check-walk that re-inferred each argument's
subtree AND recursed into it — quadratic in expression nesting. Fold the
diagnostic emission into `infer` itself (gated by a checking? flag, off
during the optimization fixpoint): one O(n) walk that both types and
checks. Removes check-walk entirely; check-form now drives infer.
This is a cleanup and removes the deep-nesting blowup, but it does NOT make
warn-by-default cheap: on a real 360-line file the checker still adds ~2.6x
compile time (277ms -> 720ms). That cost is the structural inference pass
itself, which checking inherently requires — not redundancy. A cheap
default-on path would need either piggybacking on the inference direct-link
already runs, or a lighter scalar-only checker inference. Gate green,
type-check tests pass.
rewrite-message assumed janet's BINARY arithmetic dispatch error shape
("could not find method :+ for 1 or :r+ for "a""). Unary inc/dec/- on a
non-number produce "could not find method :+ for "x"" — no "or :r" clause
— so orpos was nil and the reporter itself threw "could not find method :+
for nil", burying the real error. Handle the unary form. Found auditing
the RFC 0006 checker's default (checker-off) path. Regression row in
cli-test.
RFC 0006 error reporting wanted file:line:col but IR nodes carried no
position, so diagnostics read only "type error in <ns>: <msg>". Now:
type error /tmp/scene.clj:5:5: `inc` requires a number, but argument 1 is a string
The reader records each LIST form's absolute start offset in a table keyed
by form identity (lists are fresh arrays, never interned), gated behind a
flag the loaders enable only when JOLT_TYPE_CHECK is on — zero cost off.
Keying by identity makes positions survive macroexpansion exactly when the
user's own sub-form is spliced through, and absent for macro-synthesized
structure: a `(inc :k)` written inside `(when c ...)` reports at its own
line, never at the expansion's generated if/do.
The analyzer stamps the offset onto :invoke nodes (form-position host
contract fn); the checker carries it into each diagnostic as :pos; the
loaders stash the file's source + path on the env (save/restored across
nested requires); backend/type-check! converts offset -> line:col via the
reader's line-col and renders the RFC format. Falls back to the ns when no
position is available (synthetic forms), so it is never worse than before.
Gate green, conformance 335/335, suite 4718, runtime bench even (positions
are compile-time only; off by default).
The success checker fired only against core-fn error domains (stable, not
redefinable). This adds reporting of a call that passes a provably-wrong
type to a USER fn whose body requires otherwise — e.g. a fn that only does
arithmetic on a param, called with a string.
As check-walk sees defs it registers each non-redefinable single-fixed-arity
user fn's {:params :body} in module state (user-sig-box, accumulating across
forms like rtenv-box — a def must precede its call). At a call site (strict
mode only) the body is re-checked with ONE parameter bound to its concrete
argument type, others :any; if that produces a diagnostic the all-:any body
did not, the argument alone is provably wrong and the call is reported.
Monotonic — binding a concrete type can only add error-domain hits — so still
no false positives. A cycle guard (checking-box) terminates mutual recursion.
Gated behind JOLT_TYPE_CHECK_USER (orthogonal to the warn/error level)
because it rests on the closed-world assumption, weaker than the core-fn
case. check-form gains a strict? arity; the default path is unchanged and
user-fn code runs only when the checker is enabled. ^:redef/^:dynamic and
multi/variadic fns are not registered (their body is no stable requirement).
Gate green, suite 4718, conformance 335/335.
The success checker (RFC 0006) used to lose differing if-branches to :any
and accept the use. (inc (if c "a" :k)) typed the if as :any — sound but
imprecise, since the value is provably {:str | :kw}, every member of which
is in inc's error domain.
Adds {:union #{T...}} to the lattice: join-t forms a scalar union of
differing branches instead of collapsing to :any, capped at 4 distinct
scalars (the member space is the five scalar tags, so the lattice stays
finite and the inter-procedural fixpoint still terminates). The checker's
not-number?/not-seqable? report a union only when EVERY member is in the
error domain — any valid member accepts the call, so still no false
positives. type-name renders "a string or a keyword".
Unions are scalar-only and carry no :struct/:vec/:set key, so every
structural predicate already treats them as opaque — specialization sees
them exactly as :any and codegen is unchanged. Gate green, suite 4718,
conformance 335/335, bench even.
Reuse the structural inference from RFC 0005 as a loose type checker. It reports
a core-fn call only when an argument's inferred type is concrete and lies in
that op's throwing error domain, and accepts everything ambiguous (:any, a
union that joined to :any, :truthy). By construction it never produces a false
positive: a correct program has nothing to report even in error mode.
The curated error-domain table starts with the clearest throwing cases:
arithmetic on a provable non-number, and count/first/rest/next/seq/nth on a
provable non-seqable scalar. Lenient operations like (get 5 :k) and (:k 5),
which return nil rather than throw, are deliberately not listed.
Checking is decoupled from specialization: it runs whenever JOLT_TYPE_CHECK is
warn or error, regardless of :inline?, reading the knob at compile time so no
rebuild is needed. warn prints to stderr, error fails the form's compilation,
off (the default) skips it entirely. Core init stays clean under the flag.
jolt-y3b
Replace the ad-hoc inference lattice (a flat :struct-map tag plus {:vec ELEM})
with one recursive structural type: {:struct {field -> T}}, {:vec T}, {:set T},
scalar tags, and :any. A keyword lookup now returns its field's type, so nested
access like (:r (:direction ray)) is typed end to end and drops its guard. join
is field-wise and element-wise with a depth cap of 4 so the inter-procedural
fixpoint still terminates.
The back end honors a struct hint on any subject node, not just locals, so an
inferred field type on a nested lookup specializes. The orchestrator's fixpoint
joins through the portable join-types so compound types no longer collapse to
:any.
Ray tracer goes 12.8s to 11.0s with no hints, matching the explicit ^:struct
version (10.9s). Render checksum unchanged (1915337), full gate green,
conformance x3 modes pass.
jolt-5uj
0005 proposes replacing the ad-hoc inference lattice with one recursive
structural type (a struct carries its field types, a vector its element type,
recursively), so a lookup returns its field's type and nested access is typed
end to end. It unifies :struct tracking with field tracking, subsumes the
current inference phases, and is the soft-typing (HM + a dynamic top) design:
structural types + core-fn type schemes, solved by lattice join with :any as
top instead of unify-or-fail. Includes the depth cap for termination and an
explicit design-problems section.
0006 (follow-up, depends on 0005) reuses the inference as a loose type checker
in the success-typing discipline (Dialyzer): report only PROVABLY-wrong code
(a concrete type in an operation's throwing error-domain), accept everything
ambiguous, never a false positive. Curated error-domain table, strictness
levels (off/warn/error), clear located messages, and the soundness boundaries
(closed-world, macros, unions).
The inference now tags a :local it proved to be a vector with :hint :vector, and
the back end specializes (count v) -> pv-count (skipping core-count's dispatch
chain) and the 3-arg (nth v i default) -> pv-nth. The 2-arg nth is deliberately
NOT specialized: pv-nth returns nil out-of-bounds where Clojure nth throws.
Sound, conformance 335/335 x3 and full jpm test pass; type-infer-phase2-test
pins the specialization and the 2-arg exclusion.
Extends the inference lattice with a parametric vector type {:vec ELEM} and
threads element types through the program:
- vector literals, conj/into, and range produce element-typed vectors;
- reduce/map/mapv/filter/filterv seed their closure's element (and reduce's
accumulator) param, so a lookup inside the closure over a vector-of-structs
specializes (the HOF-element-awareness piece);
- a var reference carries a VALUE type — a fn var is :truthy (non-nil, sealed
root), a def var carries its inferred init type (e.g. a color table is
{:vec :struct-map}); element-returning fns (rand-nth/first/nth/...) yield the
collection's element type. These let the dynamically-built scene's sphere
maps type as structs.
The inter-procedural fixpoint now also infers non-fn def value types, and the
recompile re-emits the WHOLE unit callee-first (reverse-topological) so a
caller re-embeds its recompiled, now-specialized callees and a call site
compiled after the pass links the whole chain.
Result on the ray tracer (no hints): the chain closes — hittables infers to
{:vec :struct-map}, hit-sphere's hittable param to :struct-map — and the render
goes 13.1s -> 12.8s. That is only ~3%, far short of the explicit hint's 1.22x.
The remaining gap is nested field access: a lookup RESULT like (:direction ray)
is :any, so (:r (:direction ray)) stays guarded, and the vec3 fns (called with
such values) can't be typed struct. The hint asserts the vec3 params directly
and propagates through inlining; matching it needs field-shape types
(ray.direction : vec3, vec3.r : number) — a structural extension (Phase 4).
Sound: a seeded full render produces an identical checksum (1915337);
conformance 335/335 x3 and the full jpm test pass; type-infer-phase3-test pins
the element-typing + HOF mechanism. Phase 2 (vector nth/count specialization)
was deprioritized — it is orthogonal to this benchmark.
Closed-world (optimization mode): after a unit loads, infer-unit! runs a
whole-unit fixpoint over the call graph and recompiles. A fn's param types are
the lub of its in-unit call-site arg types; its return type is the lub of its
tail positions; iterated to a least fixpoint. Param types are RECOMPUTED FRESH
each iteration (not accumulated) because :any is the lattice top — joining an
early-iteration :any would poison the result permanently. Closures inherit the
enclosing tenv so captured locals keep their types (their own params shadow to
:any). A fn whose var escapes as a VALUE keeps :any params (its callers aren't
all visible). Each fn is then re-inferred with its param types seeded and
re-emitted; recompiled bodies are semantically identical, so correctness holds
regardless of order. Sound under source distribution + whole-program compile
(the consumer compiles all call sites together).
Plumbing: the portable pass (jolt.passes) gained inter-procedural primitives —
set-rtenv!, infer-body (types a body, collects its call sites), reinfer-def
(seeds param types), and escape tracking. The back end stashes each
single-fixed-arity defn's :def IR (:infer-ir); the evaluator triggers
infer-unit! after a unit loads (via an env hook, opt mode only).
Result and honest finding: the fixpoint correctly types scalar-flowing params
(ray-cast/hit-all/hit-sphere all get the ray param as :struct-map, no hint),
but the ray tracer does NOT speed up — its dominant lookups are on `hittable`,
the element of the `hittables` vector threaded through `reduce`, which stays
:any. Typing it needs collection-element types (vector<struct>) plus HOF-element
awareness (knowing reduce applies the closure to elements), which is beyond
inter-procedural param inference. The explicit ^:struct hint reaches it (it
types the reduce closure param directly), which is why the hinted run is 1.22x.
Verified: conformance 335/335 x3, full jpm test; new type-infer-phase1-test
pins the fixpoint, the escape gate, the seeded re-inference, and correctness.
A forward, soft-typing-style pass (simplified HM: monovariant, never-fails,
lattice top = :any) in jolt.passes, run after the inline/scalar-replace
fixpoint when the optimization mode is on. It types expressions from literals
and arithmetic, flows the type through let bindings, and joins at if-branches.
Where a keyword-lookup subject is PROVEN to be a plain struct map it sets
:hint :struct (the same channel a manual hint uses, so the back end drops the
:jolt/type guard); where the type is :any it leaves the dynamic guard in place.
Sound by construction: a concrete type is assigned only when proven (scalar
keys with non-nil/non-false values for a struct-map), so a wrong bare get can't
happen. This is the foundation; on its own it mostly overlaps Route 1
scalar-replacement (which already eliminates non-escaping let-bound maps), so
its standalone win is small. Phase 1 (inter-procedural) is where escaping
params get typed.
Verified: conformance 335/335 x3, full jpm test; new type-infer-test pins the
flow rules and the sound :any fallback (cases force the map to escape so the
test isolates inference from scalar-replacement).
Builds on the ^:struct keyword-lookup hint:
- ^TypeName for records. A tag naming a defrecord/deftype now resolves to the
struct fast path: record instances are tables tagged :jolt/deftype (not
:jolt/type), so a raw keyword get is correct for them. A new host contract fn
record-type? detects a record by its ->Name constructor; a non-record tag
(^String, ^long, ...) is ignored, as before.
- (get m :k) and (get m :k default) now get the same inlined keyword lookup as
(:k m): the representation guard fast path when unhinted, and the bare get
when the subject is ^:struct/^Record. A variable/number/string key still
falls through to core-get. The two call shapes share one emitter
(emit-kw-lookup).
- JOLT_CHECK_HINTS=1 turns a violated hint into a clear runtime error (naming
the local and key) by keeping the guard and throwing on the tagged arm. It is
off by default with zero cost to normal builds (a hinted lookup still emits a
bare get), and is part of the image-cache fingerprint. This is the answer to
"a lying hint is silent": opt into checking during development.
- Docs: RFC 0004 records the design, soundness contract, and measurements; the
reader spec gains S12b (hints are semantically transparent; jolt recognizes
^:struct and ^Record as lookup-optimization assertions).
There is no Clojure keyword equivalent for "plain map / fast keyword access"
(Clojure hints are class names), so ^:struct stays a jolt-specific flag,
analogous to ^:dynamic.
Verified: conformance 335/335 in all three modes and the full jpm test pass; a
seeded ray-tracer render is byte-identical hinted vs unhinted; the struct-hint
test covers record hints, the get-form, inline propagation, and the checked-mode
error. Full render with hints holds at 13.3s -> 10.9s (1.22x).
A constant-keyword lookup (:k m) currently emits a guarded form,
(if (get m :jolt/type) (core-get m k) (get m k)), to tell a plain struct
(raw get is correct) from a phm/sorted/transient (needs core-get). On a
struct that guard is a second get, so the lookup costs ~36ns where a bare
get is ~20ns. Profiling the ray tracer (jolt-dad) showed keyword lookups are
~50% of a render and the guard is the only avoidable part, but dropping it
needs to know statically that the subject is a plain struct.
Type hints are exactly that information, and jolt already parses them and
otherwise ignores them. This wires one through: a local hinted ^:struct
asserts a plain struct/record map, so a (:k local) lookup on it skips the
guard and emits a bare get. The hint rides on the binding symbol into the
analyzer, which records it per-local and attaches it to :local IR nodes; the
back end reads it on the lookup subject. It also propagates through inlining:
when the inliner let-binds a non-trivial arg to a fresh local, it carries the
called fn's param hint onto that local, so lookups inside the spliced body
keep the bare path. This is a programmer assertion, like a Clojure type hint
(an inaccurate hint just makes the raw get return the wrong value, the same
contract as a wrong ^String), so it stays opt-in and off by default.
On the ray tracer (with inlining on) this is 13.3s to 10.9s, 1.22x, taking it
to 7.8x JVM from 9.4x after the inline pass. The unhinted path emits identical
code (the fast arm is just factored out), so nothing changes without hints.
Verified: a seeded full render produces an identical checksum hinted vs
unhinted; conformance 335/335 in all three modes and the full jpm test pass;
new test/integration/struct-hint-test.janet pins the guard removal, the
inline propagation, and that an accurate hint is correctness-preserving.
Adds two IR passes to jolt.passes that run when a unit opts into
direct-linking (JOLT_DIRECT_LINK=1, off by default). The inline pass splices
small direct-linked fns at their call sites, copy-propagating trivial args so
that scalar replacement can then see map literals across the call boundary.
Scalar replacement is AOT escape analysis: a map allocation whose only use is
constant-keyword lookup is dropped and each (:k m) is replaced with the value
at :k, both for a literal lookup subject and for a non-escaping let-bound map.
Inlining and scalar replacement iterate to a capped fixpoint, since inlining
exposes literals that scalar replacement then collapses.
The back end stashes the body IR of each single-fixed-arity defn on its var
cell (inline-stash!), and the portable pass reads it through two new jolt.host
contract fns (inline-enabled?, inline-ir). Inlining is gated on :inline?, which
is off for all of init so core and the self-hosted compiler compile exactly as
before (const-fold only); api/init and main re-read JOLT_DIRECT_LINK so the
flag works both for a freshly built context and for the build-time-baked one in
the shipped binary.
Only inline-safe targets are spliced: a single fixed arity, no recur/loop/fn/
try crossing the boundary, within a size budget, a closed body (no free locals
beyond the params, so a self-recursive fn's name reference can't dangle), and
not ^:redef / ^:dynamic. Bodies are fully alpha-renamed so no spliced name can
collide with a caller local.
On the ray tracer this is 15.3s -> 13.0s (1.18x). The ceiling is honest: that
workload's cost is dominated by lookups on maps that genuinely escape (rays,
hits, materials) and by dynamic dispatch (the reduce closure, the :scatter fn),
which escape analysis cannot remove. On allocation-bound code where the
temporaries are local it is far larger: a vec3 reflect+dot loop goes 9.3s ->
0.38s (25x), with the loop body reduced to pure arithmetic.
Verified: full jpm test passes (inline off, no regression); conformance 335/335
in all three modes and the clojure-test-suite both pass with inline on; new
inline-sra-test pins the transform and its semantics.
jolt-p3c: Clojure evaluates map-literal entries left to right, but the
reader represented map forms as bare janet structs, so entries ran in
hash order. The reader now carries [k v ...] source order out-of-band —
on a struct PROTOTYPE (keys/kvs/length ignore protos, so macros that
get/keys literal map forms see no change; jolt-equal? was already
structural) and as a plain field on the phm rep (nil key/value). The
analyzer (form-map-pairs), the interpreter's map eval, both
syntax-quote walks, and core-sqmap (the lowered `{...} builder — the
array-map case, where Clojure also preserves insertion order) all honor
it, so the order survives macroexpansion in both modes.
jolt-507 root-caused: the parked inline put a LOCAL in janet call-head
position for the first time, and janet resolves head symbols against
the macro table before lexical upvalues — clojure.core/repeat's
self-name local expanded as janet's (repeat n & body) macro, compiling
the self-call into a countdown loop returning nil. Everything in the
issue (interpose, interleave) traced to that one name collision. The
emitter now rebinds local callees to reserved _fp$ symbols (argument
positions never consult the macro table), and the inline — direct
calls for function locals, jolt-call only for IFn-collection
leftovers — lands. Spec rows pin locals named repeat/seq/with called
in head position.
Gate green, suite 4718 steady, bench even with main.
jank's ray tracer benchmark (examples/ray-tracer) drops from 165.6s to
15.8s per render (10.5x) — from 118x JVM Clojure to 11x. The changes
mirror the optimizations in jank's June 2026 post, adapted to the
janet backend (jolt-4vr, jolt-h79):
- (:kw m) emits an inline lookup instead of variadic jolt-call ->
core-get's predicate chain. The guard is (get m :jolt/type): janet
compiles get to an opcode (~17ns) where a struct? cfunction call
costs ~85ns/lookup. :jolt/type is reserved (the reader rejects it in
map literals) and every table rep that must not be raw-indexed
carries it — phm tables now tagged too — so tagged values route to
core-get and everything else gets janet get, which matches core-get
for keyword keys on structs/records/nil/arrays. 929ns -> 90ns.
- {:k v ...} literals with scalar const keys emit let-bound values, an
`and` truthiness test (pure branch opcodes), and a native (struct ...)
call instead of variadic build-map-literal + runtime kv re-scan. nil
or false values fall back to build-map-literal, which keeps Clojure's
nil-entry semantics via the phm rep. 890ns -> 246ns.
- native-op additions: min/max (janet's are variadic with the same
numeric semantics), nil?/some? lowered to janet's fastfun = / not=
against nil, and not.
- clojure.math (Clojure 1.11) installed as a namespace whose vars hold
janet's math natives directly, so calls direct-link. Math/sqrt-style
interop stays in the frozen interpret-only punt set (~5us/call); this
is the compiled route (~30ns).
- reduce over pvec/tuple/array iterates indexed in place — it was
copying the whole pvec into a fresh array on every reduce call — and
stops at `reduced` instead of scanning the tail.
- interpreter coll-lookup gains the sorted-coll arm: (:k (sorted-map ..))
was nil in interpret mode (compiled mode had it right).
map-fastpath-spec pins keyword-invoke/map-literal semantics (16+15
rows incl. nil-value maps, records, sorted, vectors-of-internals) and
clojure.math (10 rows). Two pre-existing bugs found and filed while
writing it: map literals evaluate entries in reader-hash order
(jolt-p3c), and an attempted local-callee call inline that breaks
overlay lazy self-recursion is parked with a repro (jolt-507).
Gate green, conformance 335/335 x3, suite 4718 steady, core bench even
with main back-to-back.
Syntax errors were positionless ('Unterminated list', no idea where).
Now they use Clojure's shape:
Error: Syntax error reading source at (src/app/syn.clj:3:8): Unmatched delimiter: ]
at src/app/top.clj:1
The reader's 15 error sites raise a {:jolt/reader-error :msg :pos}
struct carrying the byte offset (every site already had pos in scope).
The parse entry points convert offset -> line:col on demand and
re-raise the formatted message: parse-string against its own string
(no file), parse-all-positioned against the full source with the
file threaded in from the loaders — rebasing slice-relative offsets
onto the original source so positions stay absolute. No per-token
cost; nothing is tracked until an error actually happens.
Unmatched-delimiter messages match Clojure's 'Unmatched delimiter: )'
wording. cli-test rows assert positions for unterminated string/list,
unmatched delimiter through a require (composing with round 4's
'while loading' chain), and bad ## tokens. Gate green, suite 4718
steady, bench within noise.
