Some Clojure libraries loop with the Java Iterator protocol — e.g. hiccup's
iterate! does (let [it (.iterator coll)] (while (.hasNext it) (f (.next it)))).
jolt had no Iterator, so (.iterator coll) returned nil and the loop did nothing
(silently dropping content). Add an (.iterator coll) object-method that returns a
:jolt/iterator over any seqable, with hasNext/next; the collection is materialized
via core's realize-for-iteration (late-bound through the evaluator since core
loads after it, wired in api). Found while bringing up weavejester/hiccup.
Co-authored-by: Yogthos <yogthos@gmail.com>
Two things made a program run catastrophically slow to start (ring-app: 111s),
which is backwards — direct-linking should make startup FASTER (Clojure: smaller
classes, faster startup; Stalin: whole-program analysis is strictly ahead-of-
time, zero startup cost). We were doing the opposite: paying compile-time work at
every startup.
1. Decouple inference from direct-linking. Direct-linking (cheap compile-time
call resolution) was bundled with the whole inference/specialization pipeline
(inlining + scalar replacement + structural types + the per-ns re-emit
fixpoint) — the expensive part. Now a program run direct-links + uses records
by default (fast, faster calls) but the inference is opt-in via JOLT_OPTIMIZE
(or an explicit JOLT_DIRECT_LINK / the native build). ring-app: 111s -> 6s.
2. Compile dependencies once. The baked core loads in ~10ms, but a program still
re-compiled every dependency namespace (reitit, ring, selmer, honeysql, …)
from source on every run — the whole 6s. Snapshot the ctx after the require
chain to a per-project image (the same marshal/fork the core image uses),
keyed on version + entry + source roots + flags, with a per-file mtime
manifest so any edit invalidates it. First run compiles + caches; later runs
fork the image (~10ms) and skip compilation. ring-app: 6s -> ~1s warm,
competitive with the JVM. JOLT_NO_DEPS_CACHE disables.
Full gate green (all -m integration tests now exercise the cache); deps-image
invalidation verified (touching a source recompiles).
Under direct-linking a record is a Janet tuple (its shape-rec), and core-vector?
just delegated to jvec? which is true for any tuple — so (vector? a-record) and
(sequential? a-record) returned true. That broke map-destructuring of a record:
the destructure coerce treats a sequential source as & {:keys} kwargs and does
(apply hash-map x), so destructuring a record fed its entries to make-phm as a
flat kv-list and corrupted. Surfaced as reitit's router crashing on a wildcard
route ('expected integer key for tuple in range [0,5), got 5') whenever the new
direct-link default was on; minimal repro is (let [{:keys [a]} (->R ...)] ...).
Fix: core-vector? excludes shape-recs, matching Clojure (a record is not a
vector or sequential). jvec? is unchanged for internal representation dispatch.
Regression cases added to record-declared-shape-test.
Running a program is a closed world — every namespace is required, then it runs
to completion — so make it direct-link by default (inlining, record shapes, the
inference's specialization), and for a -m/-M entry auto-enable the whole-program
cross-namespace inference pass. A decomposed multi-namespace program was ~3.7x
slower than the same code in one namespace purely because per-namespace
inference can't see a caller in a not-yet-loaded namespace; this closes that for
the common case with no flags and no hints.
Interactive modes (repl, -e, nrepl-server) stay indirect/open — they have to let
you redefine vars, which direct-linking seals against. Opt-outs:
JOLT_NO_DIRECT_LINK forces the open path even for a program run (hot-reload,
runtime redefinition); JOLT_NO_WHOLE_PROGRAM keeps direct-linking but per-ns;
JOLT_DIRECT_LINK / JOLT_WHOLE_PROGRAM still force-on. Namespaces required inside
-main (after the batch pass) fall back to per-ns inference.
The success checker (RFC 0006) rides on the inference for free, but a casual
program run shouldn't spam type warnings just because it now direct-links, so its
default-on is suppressed when direct-linking was auto-enabled (:direct-link-auto?);
an explicit JOLT_DIRECT_LINK or JOLT_TYPE_CHECK still turns it on. whole-program-
test and devirt-test opt their per-ns baseline out of the new auto-default.
Docs: RFC 0005 gains 'Compilation modes and defaults' + 'Cross-namespace
inference'; RFC 0004 documents cross-ns/param hints; self-hosting-compiler and
--help updated. Full gate green.
A ^RecordType hint only resolved against the current namespace's ctor key, so a
hint naming a record defined in another namespace degraded to :any. That made a
decomposed multi-namespace program much slower than the monolith: per-namespace
inference can't see a record param's callers in other namespaces, and the
declared hint that could have typed it was dropped.
Resolution now works cross-namespace, for both record FIELD hints (defrecord)
and fn PARAM hints, in both spellings — ^Vec3 where the type is referred and
^v/Vec3 where the namespace is aliased:
- reader keeps a tag's namespace qualifier (^t/Ray -> "t/Ray", was "Ray").
- make-deftype-ctor-impl indexes each ctor closure by value; record-hint-ctor-key
resolves a hint name against the COMPILE ns (referred names live there; aliases
resolve through it) and maps the type var's root back to its home ctor key.
Using the ctor value, not the var's :ns, is what makes :refer work — :refer
re-interns a fresh var whose :ns is the referring ns.
- the analyzer captures record param hints as arity :phints [name ctor-key];
reinfer-def seeds those param types, so a record param is typed even with no
inferred caller — the open-world / cross-ns case.
Effect on the multi-namespace ray tracer: per-ns compile 30.4s -> 7.9s with
param hints, matching whole-program (8.1s) and the single-ns monolith (8.3s).
cross-ns-hints-test covers field + param hints, refer + as, and the reader tag.
direct-var? now treats a cfunction root the same as a function root, so a
call/ref to a native fn (clojure.math/sqrt et al.) embeds the value instead of
a per-call cell deref. This was the hot indirection in the ray tracer — sqrt
runs every bounce — and it applies in every direct-link build, not just
whole-program.
const-link? is new and whole-program-only: in a closed world every non-dynamic
var has a stable root, so embed it as a constant (quoted unless it's already
callable) rather than reading the cell each reference. Covers what direct-var?
can't — ^:redef vars (reloading is off under the flag), data defs, and record
type/ctor roots. Dynamic vars stay indirect; a nil (not-yet-defined) root stays
indirect and the whole-program re-emit picks it up once the root is in place.
Measured on the records ray tracer: hot-path indirect refs (sqrt + data vars)
gone; the only indirect refs left are cold defrecord self-references. whole-
program-test now also checks a ^:redef fn and a data def so the per-ns vs
whole-program comparison guards const-link soundness.
A record field can carry a type hint — ^Vec3 (a defined record type) or ^:num —
and the inference now resolves it so reading the field back yields that exact type
instead of :any. A Vec3 stored in a Ray field reads out as Vec3, so the vec ops on
field-read values prove their reads (bare-index). This is Stalin's per-slot type
sets, but DECLARED rather than inferred: the exact shape is known up front.
- deftype captures each field's :tag / :num metadata (was stripped) and passes it
to make-deftype-ctor; the ctor registers per-field tags, resolving a record-type
hint to its ctor-key (same-ns) so the inference can look it up directly.
- call-ret-type builds a record's struct type with field types resolved from the
hints, recursing into nested record types (depth-bounded for self/cyclic types).
Measured: a nested-record read loop (:r (:origin ray)) runs 1.3s with ^Vec3 hints
vs 7.1s without — 5.5x. This is the lever the ray tracer needed (vecs flow through
container fields); records without it read back as :any and stay unproven.
A protocol method call compiles to (protocol-dispatch proto method this rest) — a
runtime registry walk (type-tag -> proto -> method) on every call, ~19x a direct
call. When the inference proves the receiver (arg 0) is a known record type, the
call now resolves to a DIRECT method call at compile time, skipping the registry.
- defprotocol registers each method's var-key 'ns/method' -> [proto method] (a
ctx-capturing register-protocol-methods! emitted into the do-block); infer-unit!
feeds it to the inference via a box (like record-shapes).
- the record-ctor return type carries :type (the record tag) so the inference
knows the receiver type; the :else invoke case annotates a protocol call whose
arg0 has a known :type with :devirt-{type,proto,method}.
- emit-invoke resolves the impl via find-protocol-method at emit time and emits a
direct call to the embedded impl fn value. Unknown/polymorphic receivers (no
proven :type) fall back to the dispatch path unchanged.
Measured: removes the dispatch overhead (14.7s -> 9.3s on a 10M-call loop); the
remaining cost is the method body itself (non-inlined, unproven reads) — inlining
the resolved method is the follow-up (jolt-t6r) toward direct-call speed.
Sound under the closed-world assumption direct-linking already makes (the impl is
resolved + embedded at compile time). Adds devirt-test (subprocess: dispatched ==
devirtualized across polymorphic dispatch, unknown-receiver fallback, and
heterogeneous collections). Stalin's compile-call/callee-environment is the model.
JOLT_WHOLE_PROGRAM (requires direct-linking) defers the per-namespace inference
and runs ONE fixpoint over every user unit at once, so param types propagate
across namespace boundaries — a non-inlined fn's record params get proven from
its callers in another unit, which the per-ns pass can't see. Sound only under
the closed-world assumption (no later eval/redefinition) the flag asserts; slow,
memory-heavy builds are the documented trade-off (the reason it's opt-in).
infer-unit! now takes one ns-name OR a list; infer-program! gathers all recorded
user namespaces and runs the existing fixpoint over the union (re-emit was already
ns-agnostic — keyed by var-key, callee-first). The evaluator defers + records each
unit under the flag; run-main triggers infer-program! after all requires, before
-main. Off by default — per-ns behaviour unchanged.
Measured: a recursive (non-inlined) cross-ns record reader runs 1.66x faster
(8.9s -> 5.3s) — params proven -> bare-index reads. NOTE: small accessor fns are
INLINED cross-ns and records carry GLOBAL declared shapes, so most record reads
are already proven without this pass; the win is for non-inlined hot fns, and it's
the foundation for future whole-program work (devirtualization, unboxing).
Adds whole-program-test (subprocess soundness: per-ns and whole-program produce
identical results on a cross-ns record program).
Records (defrecord/deftype) are now shape-recs in a direct-linking unit by
default — no JOLT_SHAPE flag. A record's shape is DECLARED, so the inference
proves field reads by a lookup, not fragile shape inference, and they bare-index.
Result: ~1.4x faster than the :jolt/deftype table form on a record-heavy loop
(3.9s vs 5.5s), driven by cheaper construction + proven bare-index reads.
Two gates now:
- :shapes? — shape-recs active; records use declared-shape layout + bare
index reads. On with direct-linking (where the inference runs).
- :map-shapes? — also shape generic const-key maps. Opt-in (JOLT_SHAPE), because
shaping maps net-loses on unproven reads (measured). Records win.
- call-ret-type types a record ctor (->Name) as a struct of its declared shape,
fed from a ctx-env registry populated at deftype; field reads on the result
bare-index. (set-record-shapes!/set-map-shapes! wired through infer-unit!.)
- sidx reads the field's position from the :shape vector AS-IS (declared order
for records, str-sorted for map literals) — no re-sort — so any field order
bare-indexes correctly. The map :map case only sets :shape under :map-shapes?.
- record-shape-for interns the descriptor per (type, fields), not per type: a
record redefined with different fields now gets a fresh descriptor instead of a
stale one (fixes redef descriptor staleness; old instances stay valid).
Adds record-declared-shape-test (declared-order reads, incl. non-alphabetical
fields, through fn boundaries + protocol method bodies). Known pre-existing edge
case filed as jolt-wf4 (direct (:f (->R …)) read returns nil after a record is
redefined with different fields; let-bound read works; repros without shapes).
Measurement (vec-heavy benchmark + read-mechanism micro-benchmarks) showed that
shape-rec'ing generic const-key maps is a net loss in a bytecode VM: an unproven
field read can't beat a native Janet struct-get (one opcode), and an inline cache
doesn't transfer to a VM jolt doesn't control. The win is real only for PROVEN
reads (bare-index beats struct) — i.e. construction-heavy or inference-friendly
code, not general map-through-fn code.
So shapes are opt-in again (JOLT_SHAPE), not defaulted on in direct-link. Kept the
get-or-shape fix: the non-proven shape read now indexes inline off the descriptor
instead of calling core-get (which was ~2.5x slower per read). :shapes? is the
single opt-in gate; image cache keys on JOLT_SHAPE + JOLT_NO_SHAPE.
Next: records with declared shapes get fast field access by default (the general
case), which is where the proven-read win actually lands.
Record shape-recs now compare type-aware like the table form: eq-map-pairs
returns nil for a record so equality falls to deep=, which is type-aware via the
per-type interned descriptor. A record equals only a same-type record, never a
plain map (cross-type and record-vs-map were wrongly equal under JOLT_SHAPE).
assoc of a new (undeclared) key keeps the record type and grows a slot, matching
Clojure's record-extension semantics.
Adds test/integration/record-shape-test.janet locking in the runtime record
mechanism (tag, field access, virtual :jolt/deftype, type-preserving assoc,
dissoc demotion, type-aware equality, #ns.Type{...} printing).
Records (defrecord/deftype) become shape-recs whose descriptor also carries
:type, under JOLT_SHAPE (flag off keeps the :jolt/deftype table form). A record
is a Janet tuple [descriptor field0 ...] in declared field order, so the
positional ->Name ctor maps args straight to slots.
- record-tag unifies the type accessor over both reps; instance?/satisfies?/
protocol dispatch and the . interop path go through it.
- virtual :jolt/deftype key on record shape-recs (via shape-get) keeps every
existing (get obj :jolt/deftype) dispatch site working.
- emit-kw-lookup's non-proven fallback routes any tuple to core-get (shape
aware), not gated on compile-time JOLT_SHAPE — core is baked without the flag
but still receives user shape-recs.
- assoc keeps the type (same-shape in place, new key grows a slot Clojure-style);
dissoc of a declared field demotes to a plain map; pr prints #ns.Type{...}.
- JOLT_SHAPE added to the image-cache key (it shapes core's own compiled paths).
The inference dropped the complete :shape whenever it rebuilt a struct type
(cap) or joined two (join-t/merge-fields), so a vec3 retrieved from a container
or a fn param typed across call sites lost its layout and every field read fell
to the slow descriptor path. Two fixes:
- cap preserves :shape: capping truncates field VALUES below the depth limit but
never the key SET, so the layout is still complete. It also recurses into
fields, so a shaped value nested in a container (a vec3 inside a hit-info)
keeps its own :shape — which is what lets (:r (:normal hit-info)) bare-index.
- join-t preserves :shape when both sides are the SAME complete shape (the
merged struct has the same keys); different shapes still drop it. This carries
the shape through if-joins and the inter-procedural fixpoint's call-site joins.
Result: the ray tracer goes from 22s (R1, correct-but-descriptor-path) to 4.36s
— 2.7x FASTER than the 11.7s no-shape baseline, and ~3x the JVM (was 8.5x), with
byte-identical output. The compounding of cheaper tuple construction plus
bare-index reads across the whole render far exceeds the per-op estimate.
Gate green flag-off, suite 4718, default-path bench even, transparency intact.
Removes ALL hardcoding. Every constant-key map literal in user code becomes a
shape-rec (a Janet tuple [descriptor v0 v1 ...]); the descriptor is interned
per key SET with a single canonical key order owned by the runtime
(types/shape-sort), so every site that builds or reads a shape agrees.
Consistency is the foundation: shape-recs are made UNCONDITIONALLY for
const-key user maps (emit-map), not gated on the inference — so a value's
representation always matches what the type system claims, which was the bug
that broke the earlier generalization (a ray built as a struct but bare-indexed
as a shape). Gated on :inline? so it applies to user data only, never to core
or the compiler's own IR-node maps.
Transparency layer (the shape-rec block now lives in types.janet, reachable by
core + evaluator + backend): get, assoc, dissoc, count, contains?, map?, first,
seq, the central realize-for-iteration normalizer (keys/vals/reduce-kv), eq-map-
pairs + eq-seqable (equality), pr-render (print), jolt-call (compiled IFn), and
the interpreter's coll-lookup all handle shape-recs. A new spec
(shape-transparency-test) asserts each op matches the equivalent struct map,
including nil/false values (which shape-recs store positionally — unlike
structs).
The ray tracer renders byte-identically (mean 122.04). Gate green with the flag
off, suite 4718. NOT yet faster — every field read currently takes the
descriptor path because the inference drops the complete :shape through joins
and containers; that's Round 2 (completeness preservation). All behind
JOLT_SHAPE (off by default).
Removes the {:r :g :b} hardcoding. ANY constant key set is now a shape:
- inference: a struct type from a map LITERAL carries :shape (its canonical
str-sorted key vector — completeness); joins/access-inferred structs lack
it, so they never get a bare index. The literal node and lookup subjects
carry the shape; the back end derives the index from it.
- backend: emit-map turns any shape-tagged const-key map into a shape tuple;
emit-kw-lookup reads the field by bare index when the complete shape is
proven, else by the value's own descriptor (so a shape-rec whose :shape was
dropped by a join still reads correctly).
- runtime: core-get and core-assoc handle shape-recs.
Status: CORRECT for direct field access, container round-trips, and assoc
(minimal repros pass). NOT yet complete — the full ray tracer still hits an
uncovered path (a shape-rec reaching a map op without coverage: keys/vals/
count/seq/equality/print/jolt-call/dissoc/contains?/the interpreter's
coll-lookup all still need shape-rec branches). And the perf win needs
COMPLETENESS PRESERVATION through joins/containers (merge-fields/cap drop
:shape today, so nested vec3 access falls to the descriptor path, slower than
a struct get) — without it the general version is slower than the vec3
prototype.
All behind JOLT_SHAPE (off by default). Gate green with the flag off, suite
4718. This preserves the general design; the transparency layer + completeness
preservation are the remaining multi-session work.
Validated prototype of the hidden-class object-model change. A vec3-shaped
{:r :g :b} map literal is represented as a cheap Janet tuple [shape vb vg vr]
instead of a struct (~2x cheaper to construct); a lookup on a value the
inference PROVES is the shape reads by bare index with no runtime check.
Result on the ray tracer (direct-link): 12.3s -> 10.7s (~13% faster), with
byte-identical pixel output. The shape value flows transparently through
hit-info/ray/material containers and the colors vector; core-get handles it
(inline check, no fn call) so an unspecialized access is still correct.
Key lessons baked in: the lookup MUST compile to a bare index (a runtime
shape check, even inlined, taxed every field read and made it 2.5-3.4x
SLOWER) — so the inference gained a :shape hint (struct type with keys
exactly {:r :g :b}) that the back end turns into (in m idx). The descriptor
is quoted when embedded (its keys are a parens tuple Janet would otherwise
try to CALL).
All behind JOLT_SHAPE (off by default). Gate green, suite 4718, default-path
bench even. Scoped to the one shape; NOT yet sound in general (assumes every
vec3-shaped value is a shape-rec, true under the flag for the ray tracer) nor
fully transparent (only core-get + the inlined lookup; jolt-call/equality/
print/keys not yet covered). Those are the next steps toward a real feature.
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
Two try bugs that blocked migratus's migrate (run/table-exists? are multi-body
try/catch/finally with janet-interop jdbc calls, which punt to the interpreter):
- The interpreter's try took only form 1 as the body, dropping later body forms
before the clauses, and did not run finally on the success path when a catch
was present. Rewritten to collect every body form before the first
catch/finally and to always run finally (via defer, so it fires even if a
catch body throws).
- current-ns leaked after a caught throw from an INTERPRETED fn (it sets ns to
its defining ns and can't restore on unwind). The interpreter's try now
restores; the compiled try (emit-try) snapshots the ns at entry and resets it
in the catch via new __current-ns/__set-current-ns! core fns. Statement
values also get a :close key so with-open can close them.
With these, migratus migrate/rollback round-trips on jolt (verified end to end
against sqlite). Conformance 335/335 x3, full gate green.
methods/get-method now take the multimethod VALUE (Clojure semantics), so the
arg must resolve to compile. The isolated analyzer never defines mf, so point
these two at print-method (a real defmulti) instead.
File API (jolt-hjw): io/file and (File. …) build a tagged :jolt/file value
(instance? File true) with a full method surface (isFile/isDirectory/exists/
getName/getPath/getAbsolutePath/listFiles/toPath/delete/createNewFile/…) backed
by os/ and file/. file-seq is File-aware (leaves are File values). str/slurp/spit
coerce :jolt/file to its path. ClassLoader/getSystemClassLoader + a classloader
stub whose getResource returns nil degrade migratus's classpath lookup to the
filesystem. java.nio.file Path/FileSystem/PathMatcher are shimmed just enough for
script-excluded?'s glob (recursive * / ? matcher).
Three bugs found getting migratus's migration discovery to work:
- (assoc nil k v) returned a raw janet table, not a map, so assoc-in built tables
that count/seq rejected. Now returns an immutable map.
- methods/get-method resolved the multimethod symbol at runtime in the current
ns, so a bare multifn ref in its defining ns saw an empty table once defmethods
lived elsewhere. Now they take the multimethod VALUE and recover the var via a
registry (Clojure semantics).
- defmulti now drops a leading docstring/attr-map (migratus's multimethods carry
docstrings) instead of treating the docstring as the dispatch fn.
Conformance 335/335 x3, clojure-test-suite at baseline.
The analyzer always took (nth items 2) as the value, so (def x "doc" 42)
bound x to the docstring and dropped 42. Now it mirrors the interpreter:
when there are 4+ items and item 2 is a string, item 2 is the docstring
(attached as :doc meta) and item 3 is the value. Conformance 335/335 x3.
For the migratus next.jdbc shim (jolt-0z5):
- core.janet: __jdbc-wrap-conn / __jdbc-conn-raw / __jdbc-make-stmt builtins.
A connection is a tagged wrapper over a jdbc.core conn carrying a clj :exec
callback so the host Statement.executeBatch runs SQL without a janet->clj call.
- javatime.janet: tagged-methods for :jolt/jdbc-conn (setAutoCommit/isClosed/
close/getMetaData), :jolt/jdbc-meta (getDatabaseProductName), :jolt/jdbc-stmt
(addBatch/executeBatch/close); java.sql.Timestamp ctor -> millis.
- evaluator.janet: instance? case for Connection/java.sql.Connection so
migratus's do-commands runs SQL through its Connection branch.
Two defn/defmacro fixes found loading migratus.core (both rooted in the reader
representing ^{:map} name metadata as a with-meta form, jolt-8w2):
- defmacro special form: unwrap a with-meta name (mirrors def), and handle the
arity-clause form (defmacro name ([params] body...)) like fn/defn — a params
vector reads as a tuple, an arity clause as a list (array).
- defn overlay: pass the bare (unwrapped) name to fn while def keeps the meta.
Conformance 335/335 x3 modes.
Faithful shim of the real clojure/tools.logging public API rather than a
bespoke logger. clojure.tools.logging.impl provides the Logger/LoggerFactory
protocols (matching upstream signatures) plus a jolt stderr-backed factory;
find-factory returns it instead of probing slf4j/log4j/jul. The macro surface
(logp/logf, level macros + f-variants, log, enabled?, spy) dispatches through
those protocols. Departures forced by no-JVM: log* writes directly (no agent/
LockingTransaction), and the throwable-first-arg branch is dropped since
(instance? Throwable x) is always false for jolt exception values.
Workarounds for two jolt gaps found en route (filed as jolt-9av, jolt-6ym):
macro bodies fully-qualify impl refs because syntax-quote does not resolve ns
aliases, and def docstrings are kept in comments because (def name doc value)
is mishandled.
stderr via __eprint. trace/debug suppressed by default (impl/*level* :info).
Conformance 335/335 x3 modes.
Two provably-wrong cases the inference already has the facts for, closing the
last RFC 0006 open question:
- Calling a non-function. At an :invoke whose callee is provably :num or :str
(the only non-callable types — keywords/maps/vectors/sets are IFn), report
"cannot call a number as a function". Default level (no closed-world: the
callee type is inferred at the call site). Covers (5 1), ("hi" 0),
((+ 1 2) :k), a let-bound number, and a var holding a number (via vtype-box
in direct-link). A union is non-callable only when every member is, so
((if c 1 :k) x) is accepted (:kw is callable). Verified zero false positives
on the ray tracer, which calls maps/keywords/vectors as fns throughout.
- Wrong arity to a user fn. The registered single-fixed-arity sig (jolt-zo1)
makes a mismatched arg count provably throw; reported under the
JOLT_TYPE_CHECK_USER opt-in (same closed-world boundary; ^:redef/variadic
skipped). Caught at compile time before the runtime arity error.
Both fold into the existing infer walk, carry :pos for file:line:col, and keep
no-false-positives. Gate green, suite 4718, conformance 335/335, runtime bench
even (compile-time only).
Character/isUpperCase + isLowerCase (ASCII, on :jolt/char structs);
Thread/interrupted (false, no real threads) + Thread/currentThread stub
with a .getContextClassLoader classloader stub; Long/valueOf; java.net.URI
constructor (string round-trip); and a minimal java.util.Date / TimeZone /
SimpleDateFormat supporting the y M d H m s tokens migratus's timestamp uses,
backed by os/date (UTC default). Full gate + 3-mode conformance green.
jolt-6xk: resolve bare exception class symbols (Exception,
IllegalArgumentException, InterruptedException, Throwable) by consulting
class-ctors/class-canonical-names in the unqualified symbol-resolution
fallthrough, mirroring the qualified path. Constructors already existed
in javatime; throw/catch/.getMessage already handled the string payload.
jolt-47b: java.util.regex.Pattern statics (compile/quote/MULTILINE) that
return jolt's native :jolt/regex value so str/replace, re-matches, and
.split accept them transparently. MULTILINE maps to a (?m) prefix routed
through the regex engine's inline-flag path; the engine gains (?m) anchor
support with the non-multiline branches left byte-identical. String
methods .matches/.replaceAll/.replaceFirst added. .split dispatches on
compiled-regex via a :jolt/regex tagged-method.
Full jpm test gate green.
Update the status, strictness levels, and open questions to reflect what
landed: bounded unions (jolt-pz5), user-fn domains behind
JOLT_TYPE_CHECK_USER (jolt-zo1), precise file:line:col (jolt-fqy), and the
checker folded into one inference walk that piggybacks on direct-link
specialization (on by default there, opt-in in plain builds). Align the
error-reporting example with the actual output format.
Checking inherently needs an inference pass (~2.6x compile as a standalone
pass). But direct-link builds ALREADY run one inference pass for
specialization (run-passes' infer-top), so checking can ride along: set a
check-mode flag, turn checking? on during that existing pass, and collect
the diagnostics after — ~2% overhead measured on the ray tracer, vs 2.6x
for the separate pass.
So the checker now defaults to `warn` in direct-link builds (where it's
nearly free) and stays OFF in plain REPL/dev builds (no inference to ride,
no forced cost — opt in with JOLT_TYPE_CHECK there). JOLT_TYPE_CHECK still
overrides in both directions (off to disable, error to escalate).
It checks the POST-optimization IR, which matches what the optimized
program actually evaluates — scalar-replace only drops provably-pure code,
an accepted opt-mode divergence, so no real error is hidden. The loaders
enable position tracking whenever checking will run (env-selected or
direct-link). type-check! (the standalone pass) stays for plain builds;
both paths share report-diags!.
cli-test pins: plain build silent, direct-link warns by default,
JOLT_TYPE_CHECK=off disables. Gate green, suite 4718, runtime bench even.
The checker ran a separate check-walk that re-inferred each argument's
subtree AND recursed into it — quadratic in expression nesting. Fold the
diagnostic emission into `infer` itself (gated by a checking? flag, off
during the optimization fixpoint): one O(n) walk that both types and
checks. Removes check-walk entirely; check-form now drives infer.
This is a cleanup and removes the deep-nesting blowup, but it does NOT make
warn-by-default cheap: on a real 360-line file the checker still adds ~2.6x
compile time (277ms -> 720ms). That cost is the structural inference pass
itself, which checking inherently requires — not redundancy. A cheap
default-on path would need either piggybacking on the inference direct-link
already runs, or a lighter scalar-only checker inference. Gate green,
type-check tests pass.
rewrite-message assumed janet's BINARY arithmetic dispatch error shape
("could not find method :+ for 1 or :r+ for "a""). Unary inc/dec/- on a
non-number produce "could not find method :+ for "x"" — no "or :r" clause
— so orpos was nil and the reporter itself threw "could not find method :+
for nil", burying the real error. Handle the unary form. Found auditing
the RFC 0006 checker's default (checker-off) path. Regression row in
cli-test.
RFC 0006 error reporting wanted file:line:col but IR nodes carried no
position, so diagnostics read only "type error in <ns>: <msg>". Now:
type error /tmp/scene.clj:5:5: `inc` requires a number, but argument 1 is a string
The reader records each LIST form's absolute start offset in a table keyed
by form identity (lists are fresh arrays, never interned), gated behind a
flag the loaders enable only when JOLT_TYPE_CHECK is on — zero cost off.
Keying by identity makes positions survive macroexpansion exactly when the
user's own sub-form is spliced through, and absent for macro-synthesized
structure: a `(inc :k)` written inside `(when c ...)` reports at its own
line, never at the expansion's generated if/do.
The analyzer stamps the offset onto :invoke nodes (form-position host
contract fn); the checker carries it into each diagnostic as :pos; the
loaders stash the file's source + path on the env (save/restored across
nested requires); backend/type-check! converts offset -> line:col via the
reader's line-col and renders the RFC format. Falls back to the ns when no
position is available (synthetic forms), so it is never worse than before.
Gate green, conformance 335/335, suite 4718, runtime bench even (positions
are compile-time only; off by default).
The success checker fired only against core-fn error domains (stable, not
redefinable). This adds reporting of a call that passes a provably-wrong
type to a USER fn whose body requires otherwise — e.g. a fn that only does
arithmetic on a param, called with a string.
As check-walk sees defs it registers each non-redefinable single-fixed-arity
user fn's {:params :body} in module state (user-sig-box, accumulating across
forms like rtenv-box — a def must precede its call). At a call site (strict
mode only) the body is re-checked with ONE parameter bound to its concrete
argument type, others :any; if that produces a diagnostic the all-:any body
did not, the argument alone is provably wrong and the call is reported.
Monotonic — binding a concrete type can only add error-domain hits — so still
no false positives. A cycle guard (checking-box) terminates mutual recursion.
Gated behind JOLT_TYPE_CHECK_USER (orthogonal to the warn/error level)
because it rests on the closed-world assumption, weaker than the core-fn
case. check-form gains a strict? arity; the default path is unchanged and
user-fn code runs only when the checker is enabled. ^:redef/^:dynamic and
multi/variadic fns are not registered (their body is no stable requirement).
Gate green, suite 4718, conformance 335/335.
The success checker (RFC 0006) used to lose differing if-branches to :any
and accept the use. (inc (if c "a" :k)) typed the if as :any — sound but
imprecise, since the value is provably {:str | :kw}, every member of which
is in inc's error domain.
Adds {:union #{T...}} to the lattice: join-t forms a scalar union of
differing branches instead of collapsing to :any, capped at 4 distinct
scalars (the member space is the five scalar tags, so the lattice stays
finite and the inter-procedural fixpoint still terminates). The checker's
not-number?/not-seqable? report a union only when EVERY member is in the
error domain — any valid member accepts the call, so still no false
positives. type-name renders "a string or a keyword".
Unions are scalar-only and carry no :struct/:vec/:set key, so every
structural predicate already treats them as opaque — specialization sees
them exactly as :any and codegen is unchanged. Gate green, suite 4718,
conformance 335/335, bench even.
Reuse the structural inference from RFC 0005 as a loose type checker. It reports
a core-fn call only when an argument's inferred type is concrete and lies in
that op's throwing error domain, and accepts everything ambiguous (:any, a
union that joined to :any, :truthy). By construction it never produces a false
positive: a correct program has nothing to report even in error mode.
The curated error-domain table starts with the clearest throwing cases:
arithmetic on a provable non-number, and count/first/rest/next/seq/nth on a
provable non-seqable scalar. Lenient operations like (get 5 :k) and (:k 5),
which return nil rather than throw, are deliberately not listed.
Checking is decoupled from specialization: it runs whenever JOLT_TYPE_CHECK is
warn or error, regardless of :inline?, reading the knob at compile time so no
rebuild is needed. warn prints to stderr, error fails the form's compilation,
off (the default) skips it entirely. Core init stays clean under the flag.
jolt-y3b
Replace the ad-hoc inference lattice (a flat :struct-map tag plus {:vec ELEM})
with one recursive structural type: {:struct {field -> T}}, {:vec T}, {:set T},
scalar tags, and :any. A keyword lookup now returns its field's type, so nested
access like (:r (:direction ray)) is typed end to end and drops its guard. join
is field-wise and element-wise with a depth cap of 4 so the inter-procedural
fixpoint still terminates.
The back end honors a struct hint on any subject node, not just locals, so an
inferred field type on a nested lookup specializes. The orchestrator's fixpoint
joins through the portable join-types so compound types no longer collapse to
:any.
Ray tracer goes 12.8s to 11.0s with no hints, matching the explicit ^:struct
version (10.9s). Render checksum unchanged (1915337), full gate green,
conformance x3 modes pass.
jolt-5uj
0005 proposes replacing the ad-hoc inference lattice with one recursive
structural type (a struct carries its field types, a vector its element type,
recursively), so a lookup returns its field's type and nested access is typed
end to end. It unifies :struct tracking with field tracking, subsumes the
current inference phases, and is the soft-typing (HM + a dynamic top) design:
structural types + core-fn type schemes, solved by lattice join with :any as
top instead of unify-or-fail. Includes the depth cap for termination and an
explicit design-problems section.
0006 (follow-up, depends on 0005) reuses the inference as a loose type checker
in the success-typing discipline (Dialyzer): report only PROVABLY-wrong code
(a concrete type in an operation's throwing error-domain), accept everything
ambiguous, never a false positive. Curated error-domain table, strictness
levels (off/warn/error), clear located messages, and the soundness boundaries
(closed-world, macros, unions).
