A bundle is closed-world — everything it needs is inlined and nothing is
required afterward — so a user defn unreachable from the entry's reference
graph can be dropped. The bundler now computes reachability from main-ns/-main
plus every non-prunable form and drops dead defn/defn- by exact source span
(formatting and reader macros in the surviving code are untouched).
Conservative and sound: only plain defn/defn- are prunable; a defn is kept if
its bare or ns-qualified name appears in any kept form, the closure runs to a
fixpoint, and any use of resolve/ns-resolve/requiring-resolve/find-var/intern/
eval/load-string disables pruning entirely. A parse failure on any file also
falls back to verbatim bundling, so the command stays as robust as a plain
concatenation. defmethod/defrecord/extend bodies are non-prunable and scanned,
so a fn reached only via dynamic dispatch stays live.
New reader/parse-all-spans returns [form start end] byte offsets so the drop
is a verbatim slice, not a re-print.
30-fn library used by a 3-fn entry: bundle 1114 -> 437 bytes (61% smaller, 27
dead fns dropped), output byte-identical.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the collection-type inference proves an argument's type, number?/
string?/keyword?/record?/nil?/some? fold to a compile-time boolean. A
const-fold now runs after inference so a folded predicate propagates and
collapses any if it gates to the taken branch.
Sound by construction: only a provable answer folds, and only when the
argument is side-effect-free (a const or local) so dropping its evaluation
is a no-op. Unknown types (:any/:truthy) and impure args keep the call.
vector?/set?/map? are left out — the :vec tag conflates a real vector with
a range/seq, so vector? could be wrong.
50M-iter loop, same shape isolated with a carry-only control: number? call+
branch 5080ms, predicate folded 1365ms — matching the 1417ms control floor,
so the 3.7x is entirely the eliminated call+branch.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Reader: #() params survive syntax-quote (auto-gensym names)
#(...) named its synthesized params with bare gensyms, so a #() written inside a
syntax-quote had its params qualified to the current ns by sq-symbol — and a
qualified symbol isn't a valid fn param. hiccup's compiler emits
`(let [sb# ..] (iterate! #(.append sb# %) ..)), which broke with "Unable to
resolve symbol: ns/_NNNN".
Name the params with a trailing # (auto-gensym suffix, like Clojure's p1__N#) so
syntax-quote maps them consistently and leaves them unqualified. Harmless outside
a backtick (just a regular symbol name).
* interop: String/valueOf static + String is a CharSequence
Two interop gaps surfaced bringing up hiccup and malli:
- String/valueOf(Object): hiccup's compiler stringifies attribute values with
(String/valueOf (or arg "")). Added the static — "null" for nil, else core-str.
- (instance? CharSequence s) returned false for a string; String implements
CharSequence, and malli's :re validator gates on it before matching, so :re
schemas always failed. instance-check now answers true for strings.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Under JOLT_OPTIMIZE a -m program run inferred + specialized EVERY loaded
namespace, including every transitive dependency. On a dep-heavy app that's
prohibitive: malli-app cold-started in ~2m10s (hundreds of dep namespaces, each
run through the per-form inline + inference passes).
The closed world a whole-program pass reasons over is the APP, not its
libraries. jolt-deps now passes the project's own source roots (its deps.edn
:paths) to the runtime as JOLT_APP_PATHS. A namespace loaded from an app root
gets full optimization (and joins the one whole-program fixpoint); a dependency
namespace compiles at default cost — :inline? off for its load, so the per-form
optimize passes don't run over library code — staying direct-linked but
generically typed (the open-world default). With no app roots declared (a bare
program run, or jolt without jolt-deps) everything counts as app, so behavior is
unchanged.
malli-app JOLT_OPTIMIZE cold start: 2m10s -> 4.5s. Compute-heavy programs whose
hot code is their own namespaces (the typed ray tracer) are unaffected — their
code is app code and still fully optimized (9s/frame render). Applied at runtime
in main for the same baked-at-build-time reason as JOLT_PATH; added to the
ctx-image cache key. Help text corrected: optimization is opt-in, not default.
Co-authored-by: Yogthos <yogthos@gmail.com>
A deftype field tagged ^:unsynchronized-mutable / ^:volatile-mutable is set!-able,
but under direct-link immutable records are shape-rec tuples, so set! errored
("Can't set! field on non-deftype: tuple").
A deftype with any mutable field now opts out of the shape-rec layout and uses
the existing :jolt/deftype table form regardless of :shapes? — set! already
mutates that form and field reads route through the tagged-table path. Such a
type is also not registered as a shape, so the inference never emits a bare-index
read against the table. Immutable deftypes/records keep the fast shape-rec.
deftype extracts per-field mutability from the field metadata and passes it to
make-deftype-ctor, which picks the representation at ctor-build time.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Don't direct-link a var redefined earlier in the same unit (jolt-wf4)
defrecord/deftype expands to (do (def R (make-deftype-ctor ...)) (def ->R R) ...),
so the ->R alias references R within one compiled unit. Under direct-link a var
ref embeds the cell's root as a compile-time constant, but on a redefine R's old
root is still in place when that unit compiles — the (def R new) sibling hasn't
run yet — so ->R sealed to the stale pre-redef ctor. (defrecord R [x a])
(defrecord R [a x]) (:a (->R 10 20)) read the old [x a] layout and returned 20.
Track the vars a unit (re)defines and force their later in-unit references to the
live indirect deref. The cell is registered only after its own init is emitted,
so a recursive self-reference inside the init still direct-links (it runs after
the def completes); only sibling references after the def go indirect.
* Emit Janet's `in` as a value so a user local can't shadow it (jolt-fjb1)
The back end emits `in` to deref var cells ((in cell :root)) and index
shape-recs. It emitted the bare symbol, so a user local named `in` shadowed
Janet's builtin in the surrounding scope and the generated cell-deref called the
user's value as a function — "<table> called with 2 arguments, possibly expected
1". malli's explainer binds [value in acc], so m/explain hit this on every
schema (m/validate was unaffected — its path doesn't bind `in`).
Embed `in`'s function value at the emit sites (as jolt-call/core-get already
are); a value in head position can't be shadowed. Fixes m/explain on malli
(loaded with JOLT_FEATURES=clj so its .cljc reader-conditionals resolve).
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
eval-form treated only a reader LIST (a Janet array) as a call; a runtime-built
list — a plist or lazy-seq from cons/concat/list or ~@ (list?/seq? true but
array? false) — fell through to self-eval. So (eval (cons '+ '(1 2))) returned
the list as data instead of 3, and a macro whose output contained such a subform
left it unevaluated. Add a plist?/lazy-seq? branch that coerces to the element
array via d-realize and dispatches through eval-list; an empty list self-evals.
The analyzer already punts these forms to the interpreter (analyze's :else ->
uncompilable -> interpreter fallback), so this one interpreter branch fixes the
correctness bug across the eval and macro-expansion paths; compiling them
directly (vs punting) would be a separate perf change. Verified: conformance
355/355, syntax-quote ~@ splice, list values unchanged.
A deftype with (Object (toString [_] s)) had its toString ignored: the generic
object-methods "toString" fired in dispatch-member before the record's own
method (the record isn't a tagged shim, so that guard passed), and str rendered
the #Type{...} data repr instead of routing through toString.
- dispatch-member: a record's own method (instance/reified/protocol) now wins
over the generic object-methods table — so .toString/.equals/.hashCode on a
record use the record's definitions; plain records still reach object-methods.
- str: add a late-bound record-tostring-cb (wired per-ctx by
install-print-method-cb!, mirroring print-method-cb) that str-render-one
consults for records — a deftype with a custom toString renders via it, plain
records keep the data repr. pr-str is unchanged.
Needed by hiccup's RawString. Adds deftype-tostring-spec (.toString + str +
concatenation + a regression guard that record-less-toString keeps its repr).
walk only handled vector?/map? and fell through :else for everything else, so
postwalk over a quoted list (a plist) never touched its elements —
postwalk-replace with symbol keys silently no-op'd, which broke
clojure.template/apply-template (found during reitit work). Add list? (rebuild as
a list) and seq? (map over it) branches after the vector/map ones so concrete
collections stay authoritative. Adds walk-spec covering list/seq walking plus a
vector/keywordize regression guard and the apply-template trigger.
analyze-try assoc'd :catch-sym/:catch-body/:finally nil-when-absent, so a try
with no catch (or no finally) carried a nil-valued key — which makes the node a
phm in jolt's map representation and forces the back end to densify it
(norm-node) before reading :op. That's the map-nil-representation trap Phase 2
already cleaned up for def/fn/arity nodes. Add those keys only when the clause is
present, matching the arity :rest discipline; a try node stays a fast struct.
Behavior-invisible: emit-try reads each key with a nil-safe (node :k) and gates
on it, so an absent key and a present-nil key are indistinguishable to every
consumer. Adds ir-try-shape-test asserting the node shape across all four
try/catch/finally combinations plus end-to-end eval.
Note on scope: the plan's "delete the defensive norm-node calls" is NOT done — it
can't be. {:op :const :val nil} (e.g. (def x nil)) and nil map keys are
inherently phm, so the emit-dispatch norm-node guards a real case, not a
present-or-absent artifact. This PR removes a source of gratuitous phm nodes
rather than the densification itself. Full gate green.
Six bottom-up IR rewrites (const-fold, inline-node, subst, flatten-lets,
subst-lookup, scalar-replace) each hand-listed every op's child positions —
~250 lines of identical "recurse children, rebuild" arms that had to be kept in
sync whenever an op was added. Extract one map-ir-children into ir.clj that
knows each op's child layout; each walk keeps only its genuine specials
(const-fold's invoke/if, inline-node's invoke, subst's local/let alpha-rename,
scalar-replace's invoke/let folds) and delegates the rest.
The combinator is total over the op set, so the walks are now total too: a
couple soundly gain coverage they previously skipped (const-fold now folds
inside :try; subst-lookup now recurses :def inits, which fixes a latent dangling
ref where a dropped const-key-map binding was referenced inside a def). These
are sound — all six are result-preserving optimizations — and 3-mode conformance
+ fixpoint confirm identical program behavior.
map-ir-children is shape-preserving for :try (recurses :catch-body/:finally only
when present, never assoc's nil) so it can't turn a struct node into a phm.
Written with cond/get only, matching the passes' tier, so no new load-order dep.
Predicates (body-closed?/pure?/local-escapes?), the type-threading infer, and the
Janet backend emit stay as-is: their conservative :else defaults / [type node]
threading / host language don't fit a node-rebuilding combinator.
Adds ir-passes-test coverage for folding reaching fn/loop/try bodies. Full gate
green (conformance x3, suite >=4695/88, fixpoint stage1==2==3, inline-sra + devirt).
eval-dot copy-pasted its entire dispatch chain across the (. obj method args...)
and (. obj member) forms — string/number/object/tagged-shim lookup duplicated,
hand-synced on every interop change. Extract one dispatch-member that takes the
evaluated args plus a has-args flag. The shared head (string/number/object/
tagged) is single-sourced; the genuinely divergent tails (call form: record →
native field → coll-interop(args); bare form: zero-arg coll-interop → field /
zero-arg method) stay branched on has-args. The guards that differed between the
arms (object-methods checks table? only; tagged dispatch checks table-or-struct;
bare-form tagged dispatch requires the member present) are preserved verbatim,
keyed off has-args, so behavior is identical.
Adds a "dot dispatch arms" spec locking the divergent cases: zero-arg vs
with-arg coll-interop, record/deftype zero-arg vs with-args methods, -field
access. Full gate green.
5d: document the seed↔overlay boundary and add a drift check. core fns split
across a Janet seed (core-X, registered in core-bindings) and a Clojure overlay;
five names (char?/sorted?/sorted-map?/sorted-set?/transduce) carry a defn in
both, with the overlay copy authoritative and the seed copy internal-only. The
into-vs-transduce home asymmetry was undocumented. Adds docs/seed-overlay-registry.md,
SEED-TWIN: comments at the five seed sites, and a build-time drift check
(test/unit/seed-overlay-registry-test.janet) that recomputes the twin set from
source and fails if it diverges or a twin leaks into core-bindings.
5e: rep↔API pointer comments in pv/plist/phm/phs/lazyseq (representation lives
here; Clojure-facing ops dispatch in core_coll/core_types) and back-pointers in
core_coll. No behavior change — comments, docs, one source-analysis test.
Full gate green (suite ≥4695 pass / ≥88 clean files), drift check passes.
main.janet held ~45 lines of env-knob policy (open-mode / direct-link / optimize
/ shapes / whole-program gates) that couldn't be unit-tested without the CLI, and
two disk-image caches (api/init-cached, main/deps-image) each hand-built a
POSITIONAL "%q|%q|..." key that silently misaligned if a ctx-shaping knob was
added in only one place.
config.janet now owns:
ctx-shaping-env-vars the canonical list of env vars that shape the built ctx
ctx-cache-key a labeled key (name=value) over a prefix + every shaping
var, so adding a knob updates BOTH cache keys at once and
can't positionally alias two different builds
resolve-run-mode [open-mode? main-entry?] -> the ctx env knob map
main shrinks to: compute open-mode?/main-entry? from argv, call resolve-run-mode,
install the knobs. Both deps-image-path (main) and image-cache-path (api) build
their keys via ctx-cache-key. New test/unit/config-test.janet locks in the
run-mode cases and asserts every ctx-shaping env var participates in the key.
Scope: this is 4a + the 4b cache-key footgun fix. The optional 4b cleanup
(folding the load/save image dance + aot marshal helpers into one ctx_image
module) is left for a follow-up — it's lower value and higher blast radius.
No behavior change (cache keys now key on a superset of env vars, so at worst a
one-time cold rebuild). Gate green: conformance 355x3, clojure-test-suite 4718
pass (>= 4695 baseline), config-test, full jpm test exit 0.
* Protocol/interop fixes to run metosin/malli
Bringing up malli (schema validation) surfaced a batch of protocol and host-interop
gaps. m/validate now works across the schema vocabulary (predicates, :map incl.
nested/optional, :vector, :tuple, :enum, :maybe, :and, bounded int/string).
- extend-type and reify now accept MULTIPLE protocols in one form (each bare
symbol switches the current protocol). reify records every protocol it
implements, so instance?/satisfies? recognise all of them.
- Protocol method params support destructuring: reify/extend-type/deftype/
defrecord emit (fn ...) (which desugars patterns) instead of raw fn*.
- instance? of a PROTOCOL works like satisfies? for reify/record instances,
matching short names across qualified/bare protocol references.
- @x reads as the qualified clojure.core/deref, so it still derefs where a ns
excludes and rebinds deref (malli does). Updated reader-test + the reader
spec/grammar (S11, deref rule).
- Java collection interop on jolt collections: .nth/.count/.valAt/.get/.seq/
.containsKey route to the clojure.core equivalent (1-arg and 0-arg paths).
- java.util.HashMap capacity/load-factor constructors + .putAll.
- A class used as a value resolves to its instances' type, so Pattern -> the
regex type (malli keys class-schemas by it).
- Shims for malli's load path: LazilyPersistentVector/createOwning and
PersistentArrayMap/createWithCheck statics.
m/explain not yet working (jolt-fjb1). Full gate green.
* satisfies? recognizes reify, consistent with instance?
A reify's protocol methods are instance-local, so they aren't in the global type
registry that type-satisfies? consults — satisfies? returned false for a reify
even when it implemented the protocol. Check the protocols the reify records on
itself (the same :jolt/protocols list instance? uses), matching short names like
instance? does. Covers single- and multi-protocol reify.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add architecture refactor plan
Synthesizes a six-part architectural review into phased, gate-validated cleanup
work. Targets LLM-maintainability: one home per feature, no god-files, explicit
checked contracts, no copy-paste dispatch. No code changes yet — the plan only.
* Refactor phase 0: dead code + isolated bugs
Pure cleanup ahead of the structural phases (docs/architecture-refactor-plan.md).
No behavior change except the two bug fixes, which are covered by a regression row.
Dead code (all verified zero-reference or overridden):
- core-resolve / core-satisfies? / core-type->str seed stubs + bindings —
resolve and satisfies? are interned by install-stateful-fns! (the seed copies
were shadowed); type->str was an inert SCI stub with no callers.
- find defined twice in 20-coll.clj; the dead copy returned a plain vector
(wrong — the live def at :787 returns a real map-entry) with a comment that
contradicted it.
- mark-hint (passes.clj), phs-to-struct (phm), shape-vals / ns-imports-fn
(types) — unreferenced.
- redundant local pad2 in javatime (module-level one already in scope).
Bugs:
- File.toURL stored :url but every :jolt/url method reads :spec, so a URL from
(.toURL file) returned nil from all its methods. Now stores :spec (+ spec row).
- pl-rest had a no-op (if (plist? r) r r); collapsed to r.
- :map-shapes? was missing from the deps-image cache key — two runs differing
only in map-shapes could reuse each other's image.
Also dropped read-quote's unused pos param. Full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Under direct-linking a record is a Janet tuple (its shape-rec), and core-vector?
just delegated to jvec? which is true for any tuple — so (vector? a-record) and
(sequential? a-record) returned true. That broke map-destructuring of a record:
the destructure coerce treats a sequential source as & {:keys} kwargs and does
(apply hash-map x), so destructuring a record fed its entries to make-phm as a
flat kv-list and corrupted. Surfaced as reitit's router crashing on a wildcard
route ('expected integer key for tuple in range [0,5), got 5') whenever the new
direct-link default was on; minimal repro is (let [{:keys [a]} (->R ...)] ...).
Fix: core-vector? excludes shape-recs, matching Clojure (a record is not a
vector or sequential). jvec? is unchanged for internal representation dispatch.
Regression cases added to record-declared-shape-test.
Running a program is a closed world — every namespace is required, then it runs
to completion — so make it direct-link by default (inlining, record shapes, the
inference's specialization), and for a -m/-M entry auto-enable the whole-program
cross-namespace inference pass. A decomposed multi-namespace program was ~3.7x
slower than the same code in one namespace purely because per-namespace
inference can't see a caller in a not-yet-loaded namespace; this closes that for
the common case with no flags and no hints.
Interactive modes (repl, -e, nrepl-server) stay indirect/open — they have to let
you redefine vars, which direct-linking seals against. Opt-outs:
JOLT_NO_DIRECT_LINK forces the open path even for a program run (hot-reload,
runtime redefinition); JOLT_NO_WHOLE_PROGRAM keeps direct-linking but per-ns;
JOLT_DIRECT_LINK / JOLT_WHOLE_PROGRAM still force-on. Namespaces required inside
-main (after the batch pass) fall back to per-ns inference.
The success checker (RFC 0006) rides on the inference for free, but a casual
program run shouldn't spam type warnings just because it now direct-links, so its
default-on is suppressed when direct-linking was auto-enabled (:direct-link-auto?);
an explicit JOLT_DIRECT_LINK or JOLT_TYPE_CHECK still turns it on. whole-program-
test and devirt-test opt their per-ns baseline out of the new auto-default.
Docs: RFC 0005 gains 'Compilation modes and defaults' + 'Cross-namespace
inference'; RFC 0004 documents cross-ns/param hints; self-hosting-compiler and
--help updated. Full gate green.
A ^RecordType hint only resolved against the current namespace's ctor key, so a
hint naming a record defined in another namespace degraded to :any. That made a
decomposed multi-namespace program much slower than the monolith: per-namespace
inference can't see a record param's callers in other namespaces, and the
declared hint that could have typed it was dropped.
Resolution now works cross-namespace, for both record FIELD hints (defrecord)
and fn PARAM hints, in both spellings — ^Vec3 where the type is referred and
^v/Vec3 where the namespace is aliased:
- reader keeps a tag's namespace qualifier (^t/Ray -> "t/Ray", was "Ray").
- make-deftype-ctor-impl indexes each ctor closure by value; record-hint-ctor-key
resolves a hint name against the COMPILE ns (referred names live there; aliases
resolve through it) and maps the type var's root back to its home ctor key.
Using the ctor value, not the var's :ns, is what makes :refer work — :refer
re-interns a fresh var whose :ns is the referring ns.
- the analyzer captures record param hints as arity :phints [name ctor-key];
reinfer-def seeds those param types, so a record param is typed even with no
inferred caller — the open-world / cross-ns case.
Effect on the multi-namespace ray tracer: per-ns compile 30.4s -> 7.9s with
param hints, matching whole-program (8.1s) and the single-ns monolith (8.3s).
cross-ns-hints-test covers field + param hints, refer + as, and the reader tag.
direct-var? now treats a cfunction root the same as a function root, so a
call/ref to a native fn (clojure.math/sqrt et al.) embeds the value instead of
a per-call cell deref. This was the hot indirection in the ray tracer — sqrt
runs every bounce — and it applies in every direct-link build, not just
whole-program.
const-link? is new and whole-program-only: in a closed world every non-dynamic
var has a stable root, so embed it as a constant (quoted unless it's already
callable) rather than reading the cell each reference. Covers what direct-var?
can't — ^:redef vars (reloading is off under the flag), data defs, and record
type/ctor roots. Dynamic vars stay indirect; a nil (not-yet-defined) root stays
indirect and the whole-program re-emit picks it up once the root is in place.
Measured on the records ray tracer: hot-path indirect refs (sqrt + data vars)
gone; the only indirect refs left are cold defrecord self-references. whole-
program-test now also checks a ^:redef fn and a data def so the per-ns vs
whole-program comparison guards const-link soundness.
A protocol method call compiles to (protocol-dispatch proto method this rest) — a
runtime registry walk (type-tag -> proto -> method) on every call, ~19x a direct
call. When the inference proves the receiver (arg 0) is a known record type, the
call now resolves to a DIRECT method call at compile time, skipping the registry.
- defprotocol registers each method's var-key 'ns/method' -> [proto method] (a
ctx-capturing register-protocol-methods! emitted into the do-block); infer-unit!
feeds it to the inference via a box (like record-shapes).
- the record-ctor return type carries :type (the record tag) so the inference
knows the receiver type; the :else invoke case annotates a protocol call whose
arg0 has a known :type with :devirt-{type,proto,method}.
- emit-invoke resolves the impl via find-protocol-method at emit time and emits a
direct call to the embedded impl fn value. Unknown/polymorphic receivers (no
proven :type) fall back to the dispatch path unchanged.
Measured: removes the dispatch overhead (14.7s -> 9.3s on a 10M-call loop); the
remaining cost is the method body itself (non-inlined, unproven reads) — inlining
the resolved method is the follow-up (jolt-t6r) toward direct-call speed.
Sound under the closed-world assumption direct-linking already makes (the impl is
resolved + embedded at compile time). Adds devirt-test (subprocess: dispatched ==
devirtualized across polymorphic dispatch, unknown-receiver fallback, and
heterogeneous collections). Stalin's compile-call/callee-environment is the model.
JOLT_WHOLE_PROGRAM (requires direct-linking) defers the per-namespace inference
and runs ONE fixpoint over every user unit at once, so param types propagate
across namespace boundaries — a non-inlined fn's record params get proven from
its callers in another unit, which the per-ns pass can't see. Sound only under
the closed-world assumption (no later eval/redefinition) the flag asserts; slow,
memory-heavy builds are the documented trade-off (the reason it's opt-in).
infer-unit! now takes one ns-name OR a list; infer-program! gathers all recorded
user namespaces and runs the existing fixpoint over the union (re-emit was already
ns-agnostic — keyed by var-key, callee-first). The evaluator defers + records each
unit under the flag; run-main triggers infer-program! after all requires, before
-main. Off by default — per-ns behaviour unchanged.
Measured: a recursive (non-inlined) cross-ns record reader runs 1.66x faster
(8.9s -> 5.3s) — params proven -> bare-index reads. NOTE: small accessor fns are
INLINED cross-ns and records carry GLOBAL declared shapes, so most record reads
are already proven without this pass; the win is for non-inlined hot fns, and it's
the foundation for future whole-program work (devirtualization, unboxing).
Adds whole-program-test (subprocess soundness: per-ns and whole-program produce
identical results on a cross-ns record program).
Records (defrecord/deftype) are now shape-recs in a direct-linking unit by
default — no JOLT_SHAPE flag. A record's shape is DECLARED, so the inference
proves field reads by a lookup, not fragile shape inference, and they bare-index.
Result: ~1.4x faster than the :jolt/deftype table form on a record-heavy loop
(3.9s vs 5.5s), driven by cheaper construction + proven bare-index reads.
Two gates now:
- :shapes? — shape-recs active; records use declared-shape layout + bare
index reads. On with direct-linking (where the inference runs).
- :map-shapes? — also shape generic const-key maps. Opt-in (JOLT_SHAPE), because
shaping maps net-loses on unproven reads (measured). Records win.
- call-ret-type types a record ctor (->Name) as a struct of its declared shape,
fed from a ctx-env registry populated at deftype; field reads on the result
bare-index. (set-record-shapes!/set-map-shapes! wired through infer-unit!.)
- sidx reads the field's position from the :shape vector AS-IS (declared order
for records, str-sorted for map literals) — no re-sort — so any field order
bare-indexes correctly. The map :map case only sets :shape under :map-shapes?.
- record-shape-for interns the descriptor per (type, fields), not per type: a
record redefined with different fields now gets a fresh descriptor instead of a
stale one (fixes redef descriptor staleness; old instances stay valid).
Adds record-declared-shape-test (declared-order reads, incl. non-alphabetical
fields, through fn boundaries + protocol method bodies). Known pre-existing edge
case filed as jolt-wf4 (direct (:f (->R …)) read returns nil after a record is
redefined with different fields; let-bound read works; repros without shapes).
Record shape-recs now compare type-aware like the table form: eq-map-pairs
returns nil for a record so equality falls to deep=, which is type-aware via the
per-type interned descriptor. A record equals only a same-type record, never a
plain map (cross-type and record-vs-map were wrongly equal under JOLT_SHAPE).
assoc of a new (undeclared) key keeps the record type and grows a slot, matching
Clojure's record-extension semantics.
Adds test/integration/record-shape-test.janet locking in the runtime record
mechanism (tag, field access, virtual :jolt/deftype, type-preserving assoc,
dissoc demotion, type-aware equality, #ns.Type{...} printing).
Removes ALL hardcoding. Every constant-key map literal in user code becomes a
shape-rec (a Janet tuple [descriptor v0 v1 ...]); the descriptor is interned
per key SET with a single canonical key order owned by the runtime
(types/shape-sort), so every site that builds or reads a shape agrees.
Consistency is the foundation: shape-recs are made UNCONDITIONALLY for
const-key user maps (emit-map), not gated on the inference — so a value's
representation always matches what the type system claims, which was the bug
that broke the earlier generalization (a ray built as a struct but bare-indexed
as a shape). Gated on :inline? so it applies to user data only, never to core
or the compiler's own IR-node maps.
Transparency layer (the shape-rec block now lives in types.janet, reachable by
core + evaluator + backend): get, assoc, dissoc, count, contains?, map?, first,
seq, the central realize-for-iteration normalizer (keys/vals/reduce-kv), eq-map-
pairs + eq-seqable (equality), pr-render (print), jolt-call (compiled IFn), and
the interpreter's coll-lookup all handle shape-recs. A new spec
(shape-transparency-test) asserts each op matches the equivalent struct map,
including nil/false values (which shape-recs store positionally — unlike
structs).
The ray tracer renders byte-identically (mean 122.04). Gate green with the flag
off, suite 4718. NOT yet faster — every field read currently takes the
descriptor path because the inference drops the complete :shape through joins
and containers; that's Round 2 (completeness preservation). All behind
JOLT_SHAPE (off by default).
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
methods/get-method now take the multimethod VALUE (Clojure semantics), so the
arg must resolve to compile. The isolated analyzer never defines mf, so point
these two at print-method (a real defmulti) instead.
Two provably-wrong cases the inference already has the facts for, closing the
last RFC 0006 open question:
- Calling a non-function. At an :invoke whose callee is provably :num or :str
(the only non-callable types — keywords/maps/vectors/sets are IFn), report
"cannot call a number as a function". Default level (no closed-world: the
callee type is inferred at the call site). Covers (5 1), ("hi" 0),
((+ 1 2) :k), a let-bound number, and a var holding a number (via vtype-box
in direct-link). A union is non-callable only when every member is, so
((if c 1 :k) x) is accepted (:kw is callable). Verified zero false positives
on the ray tracer, which calls maps/keywords/vectors as fns throughout.
- Wrong arity to a user fn. The registered single-fixed-arity sig (jolt-zo1)
makes a mismatched arg count provably throw; reported under the
JOLT_TYPE_CHECK_USER opt-in (same closed-world boundary; ^:redef/variadic
skipped). Caught at compile time before the runtime arity error.
Both fold into the existing infer walk, carry :pos for file:line:col, and keep
no-false-positives. Gate green, suite 4718, conformance 335/335, runtime bench
even (compile-time only).
Checking inherently needs an inference pass (~2.6x compile as a standalone
pass). But direct-link builds ALREADY run one inference pass for
specialization (run-passes' infer-top), so checking can ride along: set a
check-mode flag, turn checking? on during that existing pass, and collect
the diagnostics after — ~2% overhead measured on the ray tracer, vs 2.6x
for the separate pass.
So the checker now defaults to `warn` in direct-link builds (where it's
nearly free) and stays OFF in plain REPL/dev builds (no inference to ride,
no forced cost — opt in with JOLT_TYPE_CHECK there). JOLT_TYPE_CHECK still
overrides in both directions (off to disable, error to escalate).
It checks the POST-optimization IR, which matches what the optimized
program actually evaluates — scalar-replace only drops provably-pure code,
an accepted opt-mode divergence, so no real error is hidden. The loaders
enable position tracking whenever checking will run (env-selected or
direct-link). type-check! (the standalone pass) stays for plain builds;
both paths share report-diags!.
cli-test pins: plain build silent, direct-link warns by default,
JOLT_TYPE_CHECK=off disables. Gate green, suite 4718, runtime bench even.
rewrite-message assumed janet's BINARY arithmetic dispatch error shape
("could not find method :+ for 1 or :r+ for "a""). Unary inc/dec/- on a
non-number produce "could not find method :+ for "x"" — no "or :r" clause
— so orpos was nil and the reporter itself threw "could not find method :+
for nil", burying the real error. Handle the unary form. Found auditing
the RFC 0006 checker's default (checker-off) path. Regression row in
cli-test.
RFC 0006 error reporting wanted file:line:col but IR nodes carried no
position, so diagnostics read only "type error in <ns>: <msg>". Now:
type error /tmp/scene.clj:5:5: `inc` requires a number, but argument 1 is a string
The reader records each LIST form's absolute start offset in a table keyed
by form identity (lists are fresh arrays, never interned), gated behind a
flag the loaders enable only when JOLT_TYPE_CHECK is on — zero cost off.
Keying by identity makes positions survive macroexpansion exactly when the
user's own sub-form is spliced through, and absent for macro-synthesized
structure: a `(inc :k)` written inside `(when c ...)` reports at its own
line, never at the expansion's generated if/do.
The analyzer stamps the offset onto :invoke nodes (form-position host
contract fn); the checker carries it into each diagnostic as :pos; the
loaders stash the file's source + path on the env (save/restored across
nested requires); backend/type-check! converts offset -> line:col via the
reader's line-col and renders the RFC format. Falls back to the ns when no
position is available (synthetic forms), so it is never worse than before.
Gate green, conformance 335/335, suite 4718, runtime bench even (positions
are compile-time only; off by default).
The success checker fired only against core-fn error domains (stable, not
redefinable). This adds reporting of a call that passes a provably-wrong
type to a USER fn whose body requires otherwise — e.g. a fn that only does
arithmetic on a param, called with a string.
As check-walk sees defs it registers each non-redefinable single-fixed-arity
user fn's {:params :body} in module state (user-sig-box, accumulating across
forms like rtenv-box — a def must precede its call). At a call site (strict
mode only) the body is re-checked with ONE parameter bound to its concrete
argument type, others :any; if that produces a diagnostic the all-:any body
did not, the argument alone is provably wrong and the call is reported.
Monotonic — binding a concrete type can only add error-domain hits — so still
no false positives. A cycle guard (checking-box) terminates mutual recursion.
Gated behind JOLT_TYPE_CHECK_USER (orthogonal to the warn/error level)
because it rests on the closed-world assumption, weaker than the core-fn
case. check-form gains a strict? arity; the default path is unchanged and
user-fn code runs only when the checker is enabled. ^:redef/^:dynamic and
multi/variadic fns are not registered (their body is no stable requirement).
Gate green, suite 4718, conformance 335/335.
The success checker (RFC 0006) used to lose differing if-branches to :any
and accept the use. (inc (if c "a" :k)) typed the if as :any — sound but
imprecise, since the value is provably {:str | :kw}, every member of which
is in inc's error domain.
Adds {:union #{T...}} to the lattice: join-t forms a scalar union of
differing branches instead of collapsing to :any, capped at 4 distinct
scalars (the member space is the five scalar tags, so the lattice stays
finite and the inter-procedural fixpoint still terminates). The checker's
not-number?/not-seqable? report a union only when EVERY member is in the
error domain — any valid member accepts the call, so still no false
positives. type-name renders "a string or a keyword".
Unions are scalar-only and carry no :struct/:vec/:set key, so every
structural predicate already treats them as opaque — specialization sees
them exactly as :any and codegen is unchanged. Gate green, suite 4718,
conformance 335/335, bench even.
Reuse the structural inference from RFC 0005 as a loose type checker. It reports
a core-fn call only when an argument's inferred type is concrete and lies in
that op's throwing error domain, and accepts everything ambiguous (:any, a
union that joined to :any, :truthy). By construction it never produces a false
positive: a correct program has nothing to report even in error mode.
The curated error-domain table starts with the clearest throwing cases:
arithmetic on a provable non-number, and count/first/rest/next/seq/nth on a
provable non-seqable scalar. Lenient operations like (get 5 :k) and (:k 5),
which return nil rather than throw, are deliberately not listed.
Checking is decoupled from specialization: it runs whenever JOLT_TYPE_CHECK is
warn or error, regardless of :inline?, reading the knob at compile time so no
rebuild is needed. warn prints to stderr, error fails the form's compilation,
off (the default) skips it entirely. Core init stays clean under the flag.
jolt-y3b
Replace the ad-hoc inference lattice (a flat :struct-map tag plus {:vec ELEM})
with one recursive structural type: {:struct {field -> T}}, {:vec T}, {:set T},
scalar tags, and :any. A keyword lookup now returns its field's type, so nested
access like (:r (:direction ray)) is typed end to end and drops its guard. join
is field-wise and element-wise with a depth cap of 4 so the inter-procedural
fixpoint still terminates.
The back end honors a struct hint on any subject node, not just locals, so an
inferred field type on a nested lookup specializes. The orchestrator's fixpoint
joins through the portable join-types so compound types no longer collapse to
:any.
Ray tracer goes 12.8s to 11.0s with no hints, matching the explicit ^:struct
version (10.9s). Render checksum unchanged (1915337), full gate green,
conformance x3 modes pass.
jolt-5uj
The inference now tags a :local it proved to be a vector with :hint :vector, and
the back end specializes (count v) -> pv-count (skipping core-count's dispatch
chain) and the 3-arg (nth v i default) -> pv-nth. The 2-arg nth is deliberately
NOT specialized: pv-nth returns nil out-of-bounds where Clojure nth throws.
Sound, conformance 335/335 x3 and full jpm test pass; type-infer-phase2-test
pins the specialization and the 2-arg exclusion.
Extends the inference lattice with a parametric vector type {:vec ELEM} and
threads element types through the program:
- vector literals, conj/into, and range produce element-typed vectors;
- reduce/map/mapv/filter/filterv seed their closure's element (and reduce's
accumulator) param, so a lookup inside the closure over a vector-of-structs
specializes (the HOF-element-awareness piece);
- a var reference carries a VALUE type — a fn var is :truthy (non-nil, sealed
root), a def var carries its inferred init type (e.g. a color table is
{:vec :struct-map}); element-returning fns (rand-nth/first/nth/...) yield the
collection's element type. These let the dynamically-built scene's sphere
maps type as structs.
The inter-procedural fixpoint now also infers non-fn def value types, and the
recompile re-emits the WHOLE unit callee-first (reverse-topological) so a
caller re-embeds its recompiled, now-specialized callees and a call site
compiled after the pass links the whole chain.
Result on the ray tracer (no hints): the chain closes — hittables infers to
{:vec :struct-map}, hit-sphere's hittable param to :struct-map — and the render
goes 13.1s -> 12.8s. That is only ~3%, far short of the explicit hint's 1.22x.
The remaining gap is nested field access: a lookup RESULT like (:direction ray)
is :any, so (:r (:direction ray)) stays guarded, and the vec3 fns (called with
such values) can't be typed struct. The hint asserts the vec3 params directly
and propagates through inlining; matching it needs field-shape types
(ray.direction : vec3, vec3.r : number) — a structural extension (Phase 4).
Sound: a seeded full render produces an identical checksum (1915337);
conformance 335/335 x3 and the full jpm test pass; type-infer-phase3-test pins
the element-typing + HOF mechanism. Phase 2 (vector nth/count specialization)
was deprioritized — it is orthogonal to this benchmark.
Closed-world (optimization mode): after a unit loads, infer-unit! runs a
whole-unit fixpoint over the call graph and recompiles. A fn's param types are
the lub of its in-unit call-site arg types; its return type is the lub of its
tail positions; iterated to a least fixpoint. Param types are RECOMPUTED FRESH
each iteration (not accumulated) because :any is the lattice top — joining an
early-iteration :any would poison the result permanently. Closures inherit the
enclosing tenv so captured locals keep their types (their own params shadow to
:any). A fn whose var escapes as a VALUE keeps :any params (its callers aren't
all visible). Each fn is then re-inferred with its param types seeded and
re-emitted; recompiled bodies are semantically identical, so correctness holds
regardless of order. Sound under source distribution + whole-program compile
(the consumer compiles all call sites together).
Plumbing: the portable pass (jolt.passes) gained inter-procedural primitives —
set-rtenv!, infer-body (types a body, collects its call sites), reinfer-def
(seeds param types), and escape tracking. The back end stashes each
single-fixed-arity defn's :def IR (:infer-ir); the evaluator triggers
infer-unit! after a unit loads (via an env hook, opt mode only).
Result and honest finding: the fixpoint correctly types scalar-flowing params
(ray-cast/hit-all/hit-sphere all get the ray param as :struct-map, no hint),
but the ray tracer does NOT speed up — its dominant lookups are on `hittable`,
the element of the `hittables` vector threaded through `reduce`, which stays
:any. Typing it needs collection-element types (vector<struct>) plus HOF-element
awareness (knowing reduce applies the closure to elements), which is beyond
inter-procedural param inference. The explicit ^:struct hint reaches it (it
types the reduce closure param directly), which is why the hinted run is 1.22x.
Verified: conformance 335/335 x3, full jpm test; new type-infer-phase1-test
pins the fixpoint, the escape gate, the seeded re-inference, and correctness.
A forward, soft-typing-style pass (simplified HM: monovariant, never-fails,
lattice top = :any) in jolt.passes, run after the inline/scalar-replace
fixpoint when the optimization mode is on. It types expressions from literals
and arithmetic, flows the type through let bindings, and joins at if-branches.
Where a keyword-lookup subject is PROVEN to be a plain struct map it sets
:hint :struct (the same channel a manual hint uses, so the back end drops the
:jolt/type guard); where the type is :any it leaves the dynamic guard in place.
Sound by construction: a concrete type is assigned only when proven (scalar
keys with non-nil/non-false values for a struct-map), so a wrong bare get can't
happen. This is the foundation; on its own it mostly overlaps Route 1
scalar-replacement (which already eliminates non-escaping let-bound maps), so
its standalone win is small. Phase 1 (inter-procedural) is where escaping
params get typed.
Verified: conformance 335/335 x3, full jpm test; new type-infer-test pins the
flow rules and the sound :any fallback (cases force the map to escape so the
test isolates inference from scalar-replacement).
Builds on the ^:struct keyword-lookup hint:
- ^TypeName for records. A tag naming a defrecord/deftype now resolves to the
struct fast path: record instances are tables tagged :jolt/deftype (not
:jolt/type), so a raw keyword get is correct for them. A new host contract fn
record-type? detects a record by its ->Name constructor; a non-record tag
(^String, ^long, ...) is ignored, as before.
- (get m :k) and (get m :k default) now get the same inlined keyword lookup as
(:k m): the representation guard fast path when unhinted, and the bare get
when the subject is ^:struct/^Record. A variable/number/string key still
falls through to core-get. The two call shapes share one emitter
(emit-kw-lookup).
- JOLT_CHECK_HINTS=1 turns a violated hint into a clear runtime error (naming
the local and key) by keeping the guard and throwing on the tagged arm. It is
off by default with zero cost to normal builds (a hinted lookup still emits a
bare get), and is part of the image-cache fingerprint. This is the answer to
"a lying hint is silent": opt into checking during development.
- Docs: RFC 0004 records the design, soundness contract, and measurements; the
reader spec gains S12b (hints are semantically transparent; jolt recognizes
^:struct and ^Record as lookup-optimization assertions).
There is no Clojure keyword equivalent for "plain map / fast keyword access"
(Clojure hints are class names), so ^:struct stays a jolt-specific flag,
analogous to ^:dynamic.
Verified: conformance 335/335 in all three modes and the full jpm test pass; a
seeded ray-tracer render is byte-identical hinted vs unhinted; the struct-hint
test covers record hints, the get-form, inline propagation, and the checked-mode
error. Full render with hints holds at 13.3s -> 10.9s (1.22x).
A constant-keyword lookup (:k m) currently emits a guarded form,
(if (get m :jolt/type) (core-get m k) (get m k)), to tell a plain struct
(raw get is correct) from a phm/sorted/transient (needs core-get). On a
struct that guard is a second get, so the lookup costs ~36ns where a bare
get is ~20ns. Profiling the ray tracer (jolt-dad) showed keyword lookups are
~50% of a render and the guard is the only avoidable part, but dropping it
needs to know statically that the subject is a plain struct.
Type hints are exactly that information, and jolt already parses them and
otherwise ignores them. This wires one through: a local hinted ^:struct
asserts a plain struct/record map, so a (:k local) lookup on it skips the
guard and emits a bare get. The hint rides on the binding symbol into the
analyzer, which records it per-local and attaches it to :local IR nodes; the
back end reads it on the lookup subject. It also propagates through inlining:
when the inliner let-binds a non-trivial arg to a fresh local, it carries the
called fn's param hint onto that local, so lookups inside the spliced body
keep the bare path. This is a programmer assertion, like a Clojure type hint
(an inaccurate hint just makes the raw get return the wrong value, the same
contract as a wrong ^String), so it stays opt-in and off by default.
On the ray tracer (with inlining on) this is 13.3s to 10.9s, 1.22x, taking it
to 7.8x JVM from 9.4x after the inline pass. The unhinted path emits identical
code (the fast arm is just factored out), so nothing changes without hints.
Verified: a seeded full render produces an identical checksum hinted vs
unhinted; conformance 335/335 in all three modes and the full jpm test pass;
new test/integration/struct-hint-test.janet pins the guard removal, the
inline propagation, and that an accurate hint is correctness-preserving.
Adds two IR passes to jolt.passes that run when a unit opts into
direct-linking (JOLT_DIRECT_LINK=1, off by default). The inline pass splices
small direct-linked fns at their call sites, copy-propagating trivial args so
that scalar replacement can then see map literals across the call boundary.
Scalar replacement is AOT escape analysis: a map allocation whose only use is
constant-keyword lookup is dropped and each (:k m) is replaced with the value
at :k, both for a literal lookup subject and for a non-escaping let-bound map.
Inlining and scalar replacement iterate to a capped fixpoint, since inlining
exposes literals that scalar replacement then collapses.
The back end stashes the body IR of each single-fixed-arity defn on its var
cell (inline-stash!), and the portable pass reads it through two new jolt.host
contract fns (inline-enabled?, inline-ir). Inlining is gated on :inline?, which
is off for all of init so core and the self-hosted compiler compile exactly as
before (const-fold only); api/init and main re-read JOLT_DIRECT_LINK so the
flag works both for a freshly built context and for the build-time-baked one in
the shipped binary.
Only inline-safe targets are spliced: a single fixed arity, no recur/loop/fn/
try crossing the boundary, within a size budget, a closed body (no free locals
beyond the params, so a self-recursive fn's name reference can't dangle), and
not ^:redef / ^:dynamic. Bodies are fully alpha-renamed so no spliced name can
collide with a caller local.
On the ray tracer this is 15.3s -> 13.0s (1.18x). The ceiling is honest: that
workload's cost is dominated by lookups on maps that genuinely escape (rays,
hits, materials) and by dynamic dispatch (the reduce closure, the :scatter fn),
which escape analysis cannot remove. On allocation-bound code where the
temporaries are local it is far larger: a vec3 reflect+dot loop goes 9.3s ->
0.38s (25x), with the loop body reduced to pure arithmetic.
Verified: full jpm test passes (inline off, no regression); conformance 335/335
in all three modes and the clojure-test-suite both pass with inline on; new
inline-sra-test pins the transform and its semantics.
