An unknown reader tag produced the reader's internal form
{:jolt/type :jolt/tagged :tag :#foo :form bar}, which tagged-literal? didn't
recognize and which leaked as a raw map when printed:
(tagged-literal? (read-string "#foo bar")) => false ; want true
(pr-str (quote [#foo bar])) => "[{:jolt/type :jolt/tagged ...}]"
Both the data path (rdr-construct-tag) and the compile path (emit-quoted) now
build a real tagged-literal for a tag with no registered reader, like Clojure's
*default-data-reader-fn*, so tagged-literal? / :tag / :form / printing all work.
clojure.edn reads raw forms through a separate __read-form-raw path and applies
:readers/:default itself, so it is unaffected.
Re-mint (backend + reader are seed sources); prelude byte-identical, image only.
make test green (selfhost holds, 0 new/stale), +2 unit rows.
Running the whole rewrite-clj test suite (159 tests) surfaced seven more bugs;
with these it passes 3377/0/0. Each is a general jolt/JVM divergence:
- *out* was pinned to the startup stdout port, so (.write *out* …) escaped a
with-out-str capture (z/print writes via *out*). It now resolves the live
current-output-port, like print/__write, so a redirect is seen.
- nth / assoc past the end of a vector or seq threw a bare Chez error (class
:object). Throw IndexOutOfBoundsException, matching the JVM.
- A number's .toString(radix) ignored the base. Render in the base, lowercase
(rewrite-clj rebuilds 0xff / 0377 / 2r1001 through it).
- A required namespace's own :as aliases leaked into its requirer: the loaded ns
form compiles while (chez-current-ns) is still the requirer, so ce-scan-requires!
registered the loaded ns's aliases under the wrong ns and clobbered a same-named
alias there. Register an (ns NAME …) form's aliases under NAME.
- A quoted collection dropped its metadata; now it keeps USER metadata (drops the
reader's :line/:column/:file), like a Clojure quoted constant.
- enumeration-seq only did (seq e); it now drives a java.util.Enumeration through
hasMoreElements/nextElement, and StringTokenizer implements them.
Regressions: corpus rows (with-out-str/*out*, nth/assoc bounds, toString radix,
quote metadata, enumeration-seq) certified against JVM; a smoke fixture for the
alias leak (a required ns's alias must not leak). tools.reader + rewrite-clj added
to docs/libraries.md. make test green.
Running the rewrite-clj test suite under jolt exposed six bugs, each fixed here:
- `for`/`doseq` `:let` bindings never went through `destructure`, so a
destructuring pattern (`:let [{:keys [y]} x]`) hit `let*` raw and failed to
compile. Emit `let`, like Clojure.
- `with-open` couldn't close a deftype/defrecord that implements a `close` method
(java.io.Closeable / AutoCloseable, e.g. tools.reader's readers) — `__close`
only knew jhost readers and map `:close` fns. Dispatch a record's `close`.
- A deftype/defrecord method param named like a field didn't shadow the field
(the field's let-binding wrapped the params). Params now shadow, as in Clojure.
- A deftype whose simple name collided with a built-in host class clobbered it in
the global ctor table, so `(java.io.PushbackReader. …)` built tools.reader's
same-named deftype. Register deftypes/built-ins by FQN, don't let a deftype
overwrite a built-in's simple name, and qualify a bare `(Name. …)` to the
deftype's FQN only in the ns that defined it.
- `clojure.walk` was lazy over a non-list seq (missing `doall`), so a walk whose
fn has side effects read stale state. Make it eager, like Clojure.
- `Character/isWhitespace` used an ASCII-only check that missed U+2028 and other
Unicode whitespace. Use the JVM's Unicode set (minus the no-break spaces it
excludes).
Regressions: corpus rows (for-let destructure, method-param shadow, walk eager,
isWhitespace), a unit row (with-open closes a record), and smoke checks (the
class-name collision, run in a fresh -e process so the deftype doesn't leak).
One divergence remains unfixed: a submatch from a losing regex alternation branch
leaks when the winning branch has a quantified group (a bug in the vendored
irregex engine, not jolt) — tracked separately.
The REPL evaluated one line at a time, so a form split across lines
(e.g. `(+` then `1 2)`) raised instead of waiting. The read loop now
accumulates lines until delimiters are balanced — skipping string,
char, regex and comment context — printing a `... ` continuation prompt
for each extra line.
Reader/runtime errors rendered as Chez's "attempt to apply
non-procedure #[chez-pmap...]" instead of their real message. Two causes:
jolt-throw raised the thrown value raw. When a throw crossed the host
`eval` boundary, Chez re-wrapped the non-condition into a compound
condition whose message extraction applies the value, losing the message
and crashing on ex-info's empty-map :data. jolt-throw now raises a
&jolt-throw condition wrapping the value; catch (lowered to `guard`),
jolt-report-uncaught and jolt-render-throwable unwrap it back via
jolt-unwrap-throw, so ex-data/ex-message and the backtrace tag survive.
Every reader/post-prelude EOF-throw site used `(empty-pmap)` (with
parens), applying the empty-map value as a procedure and crashing during
ex-info construction before jolt-throw ran. Fixed to `empty-pmap`.
Re-minted the seed; smoke 23/23, unit 574/574.
list?, ratio?, and rational? are the predicate-web members that are
genuinely safe to migrate: not extended at runtime, not on the compiler
emit/inference path, not reached by the kernel tier. They now live in the
overlay (clojure/core/20-coll.clj) built on the jolt.host tower/rep tests,
lowering to the same code the native shims did. Removed their native
definitions (predicates.ss) and, for ratio?/rational?, the now-redundant
post-prelude re-assertions. Also dropped the dead all-flonum overlay
ratio?/rational?/decimal? stubs.
The rest of the web stays native and is documented as such: map?/set?/
seq?/coll? are extended with sorted/record/lazy arms, decimal? is extended
by the optional bigdec module, integer?/float? are on the emit/inference
path, vector? is reached by the kernel-tier peek. jolt.host exposure is
therefore narrowed to just the tests these three consume (exact?,
rational-type?, cseq?, cseq-list?, empty-list?).
Numeric probe is byte-identical to pre-migration; list? correct across
list/vector/lazy/empty/cons/rest cases. Selfhost fixpoint holds, values/
unit/smoke/corpus green, bench flat within noise.
set was a native shim (apply jolt-hash-set (seq->list coll)). It is a
pure composition, so the Clojure version (apply hash-set (seq coll))
lowers to the same code. The compiler uses set, but only off the emit
path (the backend's bare-native-names def and type inference), so it can
live in the kernel tier: compiling that tier never calls set, and by the
time those callers run the tier is already bound.
This is distinct from boolean, which the backend calls for every :if
node on the emit path. Moving boolean even to the kernel tier deadlocks
(compiling the tier that defines boolean needs boolean), so boolean stays
native. Added a comment in predicates.ss recording that.
Re-mint converges in 3 passes and the benchmark suite is unchanged
within noise (collections 43.3 vs 43.1, binary-trees 367 vs 367, the
rest flat).
Profiling jolt-i5if showed <=60-bit arithmetic is already native-fast; the real
general overhead in the run/-e/-m path is var resolution. Every var reference
compiled to (var-deref ns name), which builds + hashes a fresh "ns/name" string
and does a hashtable lookup per access (~45ns). The var cell is interned and
def-var! mutates it in place, so caching the resolved cell is sound under
redefinition.
Generalize the devirt per-site cache-cell mechanism to var value references: a
ref inside a fn resolves its cell once into the def's closure, then reads it via
var-cell-deref (a field read after the first). var-cell-deref is the cell-based
var-deref — binding-aware (dynamic vars + *ns* still resolve) and lenient on an
unbound root (a forward-declared var doesn't throw, unlike jolt-var-get).
Gated by a runtime flag: ON for runtime-compiled code (compile-eval.ss), OFF for
the seed mint and AOT build (emit-image.ss) so the seed stays a byte-fixpoint --
prelude.ss is unchanged, only image.ss picks up the new backend. ~5x on a
var-ref-heavy loop (1058ms->205ms); ~1.2x on test.check (its generators are more
deftype/dispatch-bound than var-deref-bound). No C/FFI.
Corpus rows pin redefinition / dynamic binding / forward ref through a cached
ref. make test + shakesmoke green, selfhost holds, SCI 211/218, certify 0-new.
Profiling the test.check distribution/large-sample slowness (jolt-i5if): the
hot path is the SplitMix PRNG, dominated by 64-bit mix arithmetic, and the
bitwise ops (bit-and/or/xor/not, shifts) were NOT in the backend native-ops
table — so (bit-xor a b) compiled to a var-deref through the variadic overlay
(__bit-xor) instead of a direct call, the way +/-/* already emit.
Map bit-and/or/xor/not to the Chez bitwise-and/ior/xor/not primitives (inlined
to native code; a non-integer operand now errors like the JVM instead of being
silently truncated) and the shifts to a direct helper call. bit-and-not stays on
its overlay — its only Scheme impl is 2-arg, so a value-position arity-3 use
would mis-emit.
mix-64 arithmetic 2.7x faster, raw split+rand-long 2.4x, gen/vector ~1.4x. The
remaining gap is the bignum-vs-native-long floor (~20x, substrate) plus the
generator machinery (deftype/fn dispatch, separate). Corpus rows added for value
position, bit-not, apply, and a full-64-bit unsigned shift.
map/filter/remove/take/drop/concat/take-while/drop-while/mapcat/partition
built an eager-headed cseq: the first element (and the fn application) ran
at construction, so a side-effecting (map f coll) fired f immediately and
(class (map …)) was PersistentList instead of LazySeq. This diverged from
Clojure, which wraps the whole body in lazy-seq. It went unnoticed because
the conformance gate certifies values, not realization — eager and lazy
heads produce identical values — and unit.edn even baked PersistentList in
as expected. test.check's for-all-takes-multiple-expressions (which counts
side effects in a for-all body) exposed it.
Wrap each native producer's result in a lazy-seq node so the body, incl.
the first element, defers until forced — the forced cseq still has eager
heads, so reduce/count/dorun/etc. force on walk and there's no per-element
cost. dedupe's (seq coll) is moved inside its lazy-seq. A jolt LazySeq is
now recognized by coll?/empty, the analyzer's form predicates (a macro can
build its expansion with map), value-host-tags + instance? (LazySeq/ISeq/
Sequential), and reports clojure.lang.LazySeq.
Kept the native Scheme implementations rather than porting Clojure's: a
straight lazy-seq+cons port is 3x slower and Clojure's chunked fast path is
288x slower because jolt's chunk machinery is unoptimized (filed jolt-j9dz);
the wrapped natives are Clojure-lazy at native speed.
+12 corpus rows (laziness at construction, LazySeq type, both JVM-certified).
make test + shakesmoke green, selfhost holds, 0 new divergences.
Close clojure.spec.alpha's remaining gaps — its conform/explain/describe/multi-spec
suite (clojure.test-clojure.spec, multi-spec) now passes fully.
- (get reify k) / (:k reify) routes to a reify's clojure.lang.ILookup valAt. spec
reifies fspec/regex specs as ILookup and reads (:args spec) off them, so before
this instrument never saw the args spec.
- A failed numeric comparison reports the JVM class: a nil operand is
NullPointerException, a non-number is ClassCastException (was an opaque :object
condition). conform-explain checks the thrown class.
- A quoted / macro-form #inst / #uuid literal constructs its Date/UUID value, like
the JVM reader (which builds it at read time). emit-quoted was emitting the raw
tagged form, so #inst "1939" and #inst "1939-01-01T00:00:00.000-00:00" weren't =.
- An anonymous fn reports class clojure.lang.AFunction$fn (the $fn marker), so
spec's fn-sym returns ::s/unknown for it, matching the JVM's ns$fn__N.
- A fn with & {:as m} kwargs accepts a trailing map (Clojure 1.11): (f :a 1 {:b 2})
and (f {:a 1}) both bind m, by merging an odd trailing map over the pairs.
- A thread responds to .getStackTrace (empty — jolt does TCO).
clojure.test-clojure.instr does not fully pass: its ::caller assertions need the
calling fn's stack frame, which TCO erases (an inherent host divergence, like the
JVM keeping tail frames).
make test green (+4 corpus rows, 0 new divergences), shakesmoke byte-identical.
Re-mint (backend emit-quoted + the destructure macro).
- (proxy [ThreadLocal] [] (initialValue [] body)) now builds a real per-thread
store backed by a Chez thread-parameter, with a lazy initialValue; .get/.set/
.remove work. Other proxies stay nil. test.check's no-seed PRNG (next-rng) uses
one, so gen/sample and gen/generate (and everything built on them) now work.
- clojure.test/*testing-vars* (+ *report-counters*) are bound vars now, so a
defspec run through its :test metadata / default reporter doesn't hit an unbound
var.
make test green (+1 corpus row), shakesmoke byte-identical. One re-mint (proxy).
Two general fixes shaken out by clojure.test.check's own suite (its splittable
PRNG mixes 64-bit longs and binds locals named min/max).
- *unchecked-math* now wraps arithmetic a macro emits. The analyzer rewrote a
bare (+/-/*) to its wrapping unchecked-* under *unchecked-math*, but a macro's
syntax-quote produces clojure.core/* (qualified), which was skipped — so e.g.
test.check's mix-64 multiply grew to a bignum instead of a 64-bit long. The
rewrite now also fires on the clojure.core-qualified form.
- A local binding named like a bare-emitted native op no longer shadows it. ops
where native-ops maps the name to itself (+ - * / < > min max …) emit as the
bare Scheme name; a local `max` emitted the same token, so
(fn [max] (clojure.core/max …)) called the param. munge-name now prefixes such
locals, like reserved words (derived from native-ops so they can't drift).
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (analyzer + backend).
jolt modelled letfn as a special form directly, so (macroexpand-1 '(letfn …))
returned the form unchanged. Clojure's letfn is a macro that expands to letfn*,
and macroexpansion tooling (tools.macro, tools.analyzer) depends on that — its
special-form handlers key on letfn*, not letfn.
Split it the Clojure way:
- letfn* is now the special form (analyzer), taking flat name/fn-form pairs
[name1 fn1 name2 fn2 …] — the letrec :let lowering is unchanged.
- letfn is a macro (00-syntax) turning each (name [params] body*) spec into a
name + (fn name [params] body*) binding, so it expands to letfn*.
So (macroexpand-1 '(letfn [(f [x] x)] (f 1))) now yields
(letfn* [f (fn f [x] x)] (f 1)), and clojure.tools.macro passes its whole suite
(macrolet / symbol-macrolet / mexpand-all). Listed in docs + site.
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (analyzer + the letfn macro); selfhost holds.
Two general fixes shaken out by clojure/tools.macro.
- The ns macro now accepts a vector reference clause [:require …] / [:use …],
not just the list form (:require …). Clojure dispatches on (first clause) and
accepts both; jolt silently dropped vector clauses, so a ns written with them
loaded with nothing required/used (tools.macro's test ns uses [:use …]).
- clojure.lang.Compiler/specials is now a static whose keys are the special-form
symbols (matching Clojure 1.2/1.3). Macroexpansion tooling reads
(keys Compiler/specials) to know which heads not to expand.
tools.macro itself isn't fully passing yet — its mexpand-all works, but the
macrolet/symbol-macrolet tests need letfn to macroexpand to letfn* (jolt models
letfn as a special form, not a macro over letfn*), so it stays off the list.
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (the ns macro).
data.priority-map's whole suite passes (4/4). It leans on deftype/collection
interop jolt got wrong; four general fixes:
- rseq dispatches to a deftype's clojure.lang.Reversible.rseq method instead of
always demanding a vector/sorted-coll (natives-seq.ss).
- a deftype method declared at two arities from two interfaces now dispatches by
arity: the priority-map has seq[this] (Seqable) and seq[this ascending]
(Sorted), so (.seq pm false) must reach the 2-arg one. find-method-any-protocol
now matches the call's arg count via procedure-arity-mask, and a deftype's own
declared method wins over the generic collection interop in dot-forms.
- (empty x) on a deftype/record with its own empty method uses it rather than
returning {} (jolt.host/jrec-method? gate in clojure.core/empty).
- clojure.lang.Sorted (comparator / entryKey / seqFrom) works on jolt's
sorted-map/set, so subseq/rsubseq run — including the priority-map delegating
.comparator to its backing sorted-map (dot-forms.ss + host-static.ss).
Listed in docs/libraries.md + the site. One re-mint (clojure.core/empty);
everything else runtime. make test green (0 new divergences), shakesmoke
byte-identical.
Completes the JVM long-compatibility gap so clojure.test.check (and the
property-based suites built on it, e.g. data.codec) run on jolt.
A ^long is 64-bit but a Chez fixnum is only 61-bit, so the backend's fast fx
comparison / quot / min / max / inc / dec ops raised on a full-width long (one
from the PRNG or wrapping arithmetic). They now go through the jolt-l* macros
(host/chez/seq.ss): the fx fast path when the operands ARE fixnums, the generic
op otherwise — so e.g. ((fn [^long a ^long b] (< a b)) Long/MAX 1) is false, not
an error. Arithmetic +/-/* keep the raw fx ops (under *unchecked-math* they're
already the wrapping unchecked-*).
Also fixes unsigned-bit-shift-right: it was an arithmetic (sign-propagating)
shift, now a logical shift over the 64-bit two's-complement window, so
(unsigned-bit-shift-right -1 1) is 2^63-1 like the JVM.
Result: test.check 1.1.3 loads and runs (generators, quick-check, shrinking);
data.codec's base64 property suite passes (12/12 defspecs; the 2 deftests check
clojure.lang.IFn$OLLOL, a JVM primitive-fn interface, N/A). Both added to
docs/libraries.md + the site.
re-mint (backend/seed). make test green (+3 corpus rows, 0 new divergences,
numeric gate updated to the jolt-l* ops), shakesmoke byte-identical.
clojure.core's unchecked-* (and +/-/*/inc/dec under *unchecked-math*) are long
ops that WRAP on overflow; jolt's checked arithmetic is arbitrary-precision and
its unchecked-* were plain non-wrapping (+ x y), diverging from the JVM. Now they
truncate to the low 64 bits as a signed long, matching Clojure:
(unchecked-add 9223372036854775807 1) => -9223372036854775808
(unchecked-multiply 9223372036854775807 …) => 1
- host/chez/seq.ss: jolt-wrap64 + binary jolt-unc{add,sub,mul,inc,dec,neg}2 and
the variadic clojure.core/unchecked-* fns (def-var!'d in natives-seq.ss, where
def-var! is bound). The overlay's plain unchecked-* defns are removed.
- backend lng-ops: unchecked-+/-/* emit the wrapping jolt-unc* helpers (the
raising fx ops can't wrap on Chez's 61-bit fixnums); unchecked-inc/dec too.
- *unchecked-math* is honored: the analyzer reads it (jolt.host/unchecked-math?)
and rewrites +/-/*/inc/dec to their unchecked-* for the rest of a file that
(set!)s it, like the JVM.
- jolt->fx: a ^long value that overflows the 61-bit fixnum range passes through
as an exact integer instead of erroring (a full-width long from wrapping math).
Also adds Long/bitCount / numberOfLeadingZeros / reverse and Math/getExponent /
scalb (test.check's splittable PRNG uses them).
This lets clojure.test.check load and run quick-check on jolt. re-mint (analyzer/
backend/overlay are seed sources). make test green (+6 corpus rows, 0 new
divergences, numeric gate updated), shakesmoke byte-identical.
clojure.core.contracts (over core.unify) now runs its whole suite on jolt —
14/14 across contracts/constraints/with-constraints/provide tests. Two general
gaps fixed:
- Symbol and Keyword now report IFn (and Fn/Runnable/Callable) in the modeled
class hierarchy, so a (class x)-dispatched multimethod with an IFn method
matches a symbol or keyword, like the JVM (both implement IFn — they're
callable). core.contracts' funcify* dispatches on (class constraint) and a
bare predicate symbol must hit the IFn arm. Runtime, no re-mint.
- A live Var value spliced into a form by a macro (defcurry-from resolves a var
and emits (~v l r)) now compiles: analyze treats a var-cell form as a
:the-var reference by ns+name, the same node as (var ns/name), mirroring the
existing spliced-namespace (~*ns*) case. analyzer.clj + host-contract.ss,
re-mint (prelude stays byte-identical; only the analyzer image changes).
Listed in docs/libraries.md + the site.
make test green (+2 corpus rows, 0 new divergences), shakesmoke byte-identical.
A macro body can now read &form (the call form) and &env (a map of the in-scope
local symbols), like Clojure. This is what core.logic's matche/defne use to tell
a pattern symbol that names an enclosing local from a fresh pattern var — so
locals-membero and the recursive checko in `matches` now compute correctly. The
suite reaches 535/2/0 (the last two are constraint reification ORDER, where the
constraint set is right but it is spliced from a set whose iteration order differs
from the JVM — a host set-ordering divergence, not a bug).
&form/&env are clojure.core dynamic vars bound around each expander call rather
than prepended params, so the macro calling convention is unchanged and the mint
stays consistent (the seed prelude is byte-identical; only the analyzer carries
the env into form-expand-1). macroexpand-1 passes an empty env.
corpus.edn: the ~@ unquote row is now a boolean compare (a bare clojure.core/
unquote-splicing symbol evaluates to an unbound var, not the symbol).
Follow-on to the core.logic relational-engine work. These clear every crash in
core.logic's constraint-logic-programming and unifier layers (33 errors -> 0) and
most of the value mismatches; the suite goes 504 -> 523 passing assertions. All
are general gaps, not core.logic-specific.
- symbols intern their ns/name strings (JVM Symbol.intern .intern()s them): two
separately-read `?a` symbols now share one name-string object. core.logic's
non-unique lvars compare names by identity (via (str sym)), so without this a
term's lvar and a constraint's lvar built from different `?a` reads never matched
and constraints silently never fired.
- (str x) of a single arg returns its rendering directly instead of copying through
string-append, and a symbol stringifies to its (interned) name — JVM (str x) is
x.toString(). Needed for the identity comparison above.
- a clojure.core-qualified special form dispatches correctly: syntax-quote
namespace-qualifies a macro like letfn to clojure.core/letfn (matching Clojure,
where it's a macro), and the analyzer now maps that back to the special form
instead of treating it as an invoke of a nil var. core.logic's fnc/defnc emit
(clojure.core/letfn ...). Re-mint.
- (disj nil ...) is nil (JVM), instead of crashing in the set path — core.logic's
constraint store does (disj (get km v) id) where the get can be nil.
corpus.edn: 4 JVM-certified rows. make test + shakesmoke green, 0 new divergences,
self-host fixpoint holds.
Running clojure/core.logic's own suite surfaced a batch of general jolt gaps.
None are core.logic-specific; each is a language/host behavior that was wrong or
missing. With these, the core relational engine (unify, run/fresh/conde,
conso/membero/appendo, reification to _0/_1, lcons) runs; the remaining failures
are in core.logic's constraint-logic-programming and finite-domain layers
(tracked separately).
- analyzer: accept the list-member dot form (. target (method args)), sugar for
(. target method args). Re-mint.
- identical? is reference identity (eq?), not value equality. It was aliased to =,
which infinite-loops when a deftype's .equals short-circuits on (identical? this o)
(core.logic's Substitutions) and is wrong for distinct equal collections.
- jrecs use a deftype's declared hashCode/equals/equiv for map/set keying instead
of structural field comparison, so metadata-wrapped keys still match (core.logic
keys substitutions on lvar id, ignoring metadata).
- meta/with-meta dispatch to a deftype's clojure.lang.IObj meta/withMeta methods
when present, so metadata threaded through the type's own assoc/withMeta survives
(previously kept in an identity side-table the reconstructed instances didn't share).
- coll?/seqable? on a deftype require IPersistentCollection (cons) or ISeq (first);
ILookup(valAt)/Indexed(nth)/Counted(count)/Seqable(seq) alone no longer qualify,
matching the JVM.
- syntax-quote resolves a bare symbol to the compile ns's own def before
clojure.core, so a name the ns excluded and redefined (core.logic's == after
:refer-clojure :exclude) qualifies correctly in macro output.
- reader: record literals #ns.Type{...} / #ns.Type[...] expand to the map->/->
factory call.
- structmap API: defstruct/create-struct/struct-map/struct/accessor (map-backed,
insertion-ordered). Re-mint.
- .hashCode on strings/symbols (Java String.hashCode, Symbol Util.hashCombine);
Class.isInstance; java.util.Collection.contains over vector/list/set;
clojure.lang.RT/nextID and clojure.lang.Util hash/hasheq/equiv/identical statics.
corpus.edn: 8 JVM-certified rows. unit.edn: a Counted+Seqable deftype is coll?=false
(was a stale expectation encoding the old behavior).
jolt maps were HAMTs with hash iteration order; Clojure keeps small maps as
PersistentArrayMap (insertion order), converting to PersistentHashMap past a
threshold. Map literals, array-map, assoc, into/transient, merge, zipmap,
select-keys, update-keys/vals, frequencies and group-by now iterate in insertion
order for <=8 entries, matching the JVM. hash-map and >8-entry maps stay hash
order; sets stay hash order.
The pmap record gains an order field (the insertion-order key list, or #f once
hashed); the HAMT still backs the values so equality/hash/lookup are unchanged.
pmap-fold visits an array-mode map last-to-first so the runtime's cons-accumulate
idiom reconstructs insertion order without touching its many call sites, and
hash-mode output stays byte-identical; pmap-fold-fwd visits in order for the few
sites that build a value directly. Transient maps track insertion order and
promote to hash past max(8, source-count), matching TransientArrayMap.
The hash-map native-op retargets to a hash-order builder so (hash-map ...) stays
hash-ordered while {...} literals are ordered; syntax-quote builds maps via the
hash builder (Clojure expands `{...} to apply hash-map). The core overlay map
builders seed from {} instead of (hash-map) to keep order.
Threshold is 8 for any key (the keyword exception in newer Clojure isn't in
1.12.5). honeysql now passes 832/0/0; 19 JVM-certified corpus rows added.
defn- now adds :private to the var metadata (like Clojure), and ns-publics
filters those out while ns-interns/ns-map keep them — they were all the same
unfiltered scan before. A lib that introspects ns-publics (honeysql asserts
every public helper has a docstring, and that the clause set matches the public
helpers) saw the private defn- helpers and failed; now honeysql 636/8 -> 638/6
(the rest are map key-order).
clojure.edn/read built the built-in #inst/#uuid eagerly (via read-string), so a
:readers override couldn't win and #inst applied to a non-string form (aero's
#inst ^:ref […]) threw. Read the raw form instead and let edn->value route every
tag through :readers then :default then the built-in — matching clojure.edn,
where a reader from opts wins. edn->value now also converts the (recursively
converted) metadata, since the raw path skips the read-string data seam. aero
suite: 59/0/0 (full pass). clojure.edn baked, re-minted.
clojure.edn/read with :readers/:default recursed into vectors/maps/seqs but
not a constructed set, so a tagged literal in #{…} (aero's #ref in a set) kept
its raw form. edn->value now recurses into a set. clojure.edn is baked into the
seed, re-minted. Fixes aero #ref-in-set + falsey-user-return.
Two gaps reitit-core surfaced (now 322/0/1 -> 327/0/0):
- A deftype/defrecord inline method with two _ params, (m [_ _] field), read
the field as nil: mk-clause bound fields off (get _ :field) where _ was the
first param, but the second _ shadowed it. Each _ param is now renamed to a
fresh symbol so the instance is unambiguous.
- A var did not dispatch to a protocol's clojure.lang.Var extension (reitit
extends Expand to Var for a #'handler route): value-host-tags gained a var arm
(Var/clojure.lang.Var/IDeref/IFn) and host-type-set gained Var/IDeref so the
extension keys under Var.
deftype/defrecord is a seed source, re-minted.
(defn name docstring? {:k v} arglists...) and the multi-arity name+attr-map
now merge the attr-map into the var metadata like Clojure — jolt was parsing
the map out of the body and discarding it. The metadata (the name's own ^{},
the attr-map, and the docstring as :doc) is attached to the def name symbol,
which analyze-def reads and evaluates. defn is in the earliest tier, so the
macro uses only conj/assoc/meta/with-meta (not merge/last). The rare trailing
attr-map (after the last arity) is not yet handled. Fixes hiccup's defelem
meta + honeysql docstring tests.
Six correctness fixes, each a general gap (not hiccup-specific):
- deftype is not a map. jolt treated every deftype instance as a map
(map?/record?/seqable over its fields); in Clojure only a defrecord is
map-like, a bare deftype is an opaque object. defrecord now marks its type;
map?/record?/coll?/seq/empty? gate on it, while a deftype implementing a
collection interface still dispatches through its methods.
- cross-ns extend-protocol on an imported deftype. register-method built the
type tag from the *calling* ns + bare name, so (extend-protocol P Raw …) in
one ns missed a Raw value defined in another. A simple-name index resolves
the bare name to the type's real tag (local ns still wins).
- str vs print. str of a collection is its readable form (nested strings
quoted: (str ["x"]) => ["x"]); print leaves them raw. jolt defined print
as str, conflating the two. Split via a __print1 seam.
- clojure.test thrown? now honors the exception hierarchy (instance?), so
(thrown? IllegalArgumentException …) matches an ArityException subclass.
- java.net.URI is value-equal (= and hash by string form).
- clojure.walk/macroexpand-all was missing; an unresolved qualified var made
the analyzer report "Unknown class walk".
deftype/defrecord + print are seed sources, re-minted. hiccup 365->381 of its
own suite; the rest are charset-encoding / var-meta niches.
jolt's seq layer realized one element ahead of Clojure, so a side-effecting
lazy seq ran its producer too eagerly. Four changes bring it in line:
- rest is Clojure's more(): it returns the tail without realizing it. An
unforced tail (vector / string / lazy-seq cell) comes back as a deferred
seq, so (rest (iterate f x)) does not call f. next still realizes one.
- iterate applies f lazily, inside the tail thunk, so (first (iterate f x))
is x with no call to f (clojure.lang.Iterate parity).
- take realizes exactly n: the last element terminates without touching the
rest, instead of forcing one more element of the source.
- an empty realized lazy seq is still a sequence value, printing "()" not
"nil" (a JVM LazySeq is never nil).
Also: the map transducer's step fn now takes multiple inputs
([result input & inputs]) so a multi-collection transduce applies f across
all of them. Fixes medley's join/window/sequence-padded laziness and
multi-input transducer tests (now 293/293). The rest change also fixed a
latent overrun in distinct/dedupe over a map's empty tail.
iterate is a seed source, re-minted.
(type x) was jolt's internal taxonomy keyword (:string/:set/:jolt/inst), which
breaks any library dispatching a multimethod on [(type a) (type b)] against
java/clojure.lang classes (e.g. clojure.tools.logging.test's matchers). Make the
PUBLIC clojure.core/type Clojure's (or (:type meta) (class x)).
The taxonomy keyword stays the core model: natives-meta.ss keeps jolt-type and
exposes it as __type-tag, which print-method/print-dup dispatch on (so #uuid/#regex/
records still print). The JVM mapping lives in the java host layer — host-class.ss
defines the public type next to (class …), and a jinst now reports java.util.Date
(was :jolt/inst). So the core emits the taxonomy and the java layer remaps it in one
place. unit.edn's type suite updated to the class names. make test green.
Co-authored-by: Yogthos <yogthos@gmail.com>
Class/forName claimed every java.*/clojure.* name found (and any "x.y.Class"
matched the registered Class via a short-name fallback), so a library's
(class-found? "optional.Dep") feature-probe always said yes — tools.logging then
tried to build the java.util.logging / log4j backends jolt lacks and crashed.
Resolve forName by exact registry lookup + an honest prefix that excludes the
unbacked optional packages (java.util.logging, javax.management), so the probe
sees them absent and skips the backend.
class of a persistent collection / namespace now reports its JVM class name
(clojure.lang.PersistentHashSet, …Namespace, …) instead of jolt's internal :set/
:object tag, and isa? consults JVM class assignability — Object as every class's
root plus a modeled clojure.lang/java.util hierarchy — so (isa? (class x) C) and a
class-keyed multimethod dispatch like the JVM (e.g. (isa? Keyword Object) was
false). Adds the bare class tokens (Fn/Namespace/Set/…) these dispatch on.
(type x) is unchanged — it keeps jolt's documented internal-keyword form. Six
JVM-certified corpus rows. make test green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
A devirtualized protocol call resolved its impl with devirt-resolve on EVERY call
— but the tag/proto/method are compile-time constants, so the resolved fn is a
runtime constant (closed world). That per-call find-protocol-method (three
hashtable lookups) was the cost: on mono-dispatch, dispatch was ~75% of the time
(ablation: same arithmetic direct-call 166ms vs dispatch 673ms).
Resolve once. When emitting a direct-link def, each devirt site gets a fresh cache
cell, bound to #f in a let wrapping the def (so it persists across calls and is
shared by every invocation); the site resolves into it on first use ((or cell
(let ((_f (devirt-resolve ..))) (set! cell _f) _f))) and reuses it after — the
inline cache the JVM gets for free. First call still passes the real receiver, so
the Object/host-tag fallback (devirt-resolve) is unchanged.
mono-dispatch 673ms -> 214ms (~3.15x), 47.5x -> ~15x JVM, near the 166ms
direct-call floor. run-devirt.ss gains the cached-path checks (cell present, 1st
call caches + 2nd reuses, both == dispatch). make test / shakesmoke green, selfhost
holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
A record-or-nil (a protocol method whose impls return a record in one branch and
nil in another, or an `if` over a ctor and nil) now types as a NILABLE record
instead of widening to :any. A nilable record still bare-indexes its field reads
(jrec-field-at falls back to jolt-get on nil), but some?/nil? do NOT fold on it, so
a runtime guard is preserved — and inside (if (some? x) ..) / (if x ..) the then-
branch narrows x to the non-nil record, so its reads bare-index AND unbox there.
This is what lets the bounced ray type without a hint: scatter returns
ScatterResult-or-nil (Metal absorbs some rays), and the consumer reads
(:ray scattered) only under (if (some? scattered) ..). The narrowing proves
scattered non-nil there.
lattice: :nil type; :nil ∨ struct -> nilable struct, ∨ anything else -> :any;
nilability is contagious through a struct join, which also now preserves :type when
both sides agree (needed so a record ∨ its nilable self stays that record).
truthy-type?/field-type/pred-on treat a nilable struct as maybe-nil. types: nil
literal -> :nil; an `if` whose test is (some? x)/(nil? x)/x narrows the nilable
local x in the proven branch.
Ray tracer with NO hints: 38.4s -> 23.9s (~1.6x) — hit-sphere now types fully
(0 jolt-get, 57 jrec-field-at, 38 fl-ops), identical to the hand-hinted build.
run-narrow.ss gate, incl. the load-bearing check that the nil case still takes the
else branch (the guard is not folded away). make test / shakesmoke green, selfhost
holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method whose impls all return the same record type has a monomorphic
return. collect-pm-rets! scans the unit's (register-(inline-)method ..) forms,
infers each impl fn's return type, and joins them per method; call-ret-type then
types a (method recv ..) call as that record, so a field read off the result
bare-indexes — e.g. (:ray (scatter m ..)) reads off a Ray. A disagreeing impl
joins to :any and keeps the generic path.
run-protoret.ss: a method with all-record impls bare-indexes + unboxes the field
read; a mixed-return method (one impl returns a number) stays generic. make test /
shakesmoke green, selfhost holds, 0 new divergences.
Foundation for auto-typing record values that flow through protocol dispatch. Does
not yet move the ray tracer: its scatter returns ScatterResult-or-nil (Metal
absorbs some rays), and the nil widens the join to :any — typing a nullable return
soundly needs flow-sensitive narrowing (a guarded (some? x) proves non-nil), filed
separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
The whole-program fixpoint collects a self-recursive call's arg types into the
fn's own params. When a recursive call threads a param straight through unchanged
(same arg, same position — e.g. ray-cast passing `hittables` to itself), that arg's
type is the param's own current type: :any until external callers determine it. And
:any is absorbing, so collecting it pinned the param at :any forever — the type a
caller supplied (a vector of records) was lost, and the fn's field reads stayed
generic.
Skip a same-position pass-through arg in the self-recursion collection (contribute
the join identity). It can't add information — param i ⊇ param i is trivial — so
dropping it is sound; the param is still constrained by every external caller and
by any non-pass-through recursive arg. Applies to both self-recursion paths: a
`defn` recursing through its var, and a named fn literal recursing via its
self-local.
This is why ray-cast's `ray` typed (its recursion passes a fresh ray) but
`hittables` didn't (passed through). With the fix, hittables keeps its
vec<Sphere> element type, so hit-all's reduce element — and hit-sphere's reads —
type without any hint: ray tracer 38.4s -> 31.3s (~1.23x) with no annotations.
run-wp.ss: a recursive fn threading a vec param through keeps its element type.
make test / shakesmoke green, selfhost holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
A record field tagged ^double now reads back as a flonum and feeds the numeric
pass, so hintless arithmetic over those fields lowers to fl-ops — the leaf-numeric
analog of the ^Vec3 nested-field hints. Combined with the whole-program :double
param inference, a vec3-dot over a ^double-fielded record unboxes end to end with
no per-fn hints.
records.ss: a ^double field tag passes through resolution, and the ctor (and a
mutable-field set!) coerce a ^double field to a flonum — JVM primitive-field
parity (jolt returned an exact 1, not 1.0, before), and what makes reading the
field back as :double sound for an fl-op.
types.clj: field-type-from-tag maps "double" -> :double, and a keyword/get lookup
whose result is :double annotates the node :num-read :double. numeric.clj reads
that annotation and classifies the field read as a :double operand, so the
enclosing arithmetic specializes — the read itself keeps its jrec-field-at/jolt-get
emit.
run-fieldnum.ss gate: ctor coercion (int field -> flonum), field-field arithmetic
emitting fl*/fl+, and an untagged field staying generic. make test / shakesmoke
green, selfhost holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
The closed-world fixpoint (#226) flowed record types across fn boundaries; this
adds a numeric refinement so a hintless fn whose every call site passes a flonum
has its param unboxed to fl-ops, no ^double hint needed.
Lattice gains :double, a flonum refinement of :num: two doubles join to :double,
a double joined with anything else widens to :num — so a param is :double only
when every contributing value is a flonum, which is what makes the fl-op sound.
infer types a flonum literal and flonum arithmetic (+ - * / min max inc dec over
double/int-literal operands) as :double, and the fixpoint joins those across call
sites and return types like any other lattice value.
The bridge to the existing hint-directed pass is a synthetic [param :double]
nhint: wp-infer! stashes the :double params separately from the structural seeds,
and run-passes injects them as nhints before numeric/annotate, so the fl-op
emission and the exact->inexact entry coercion (a no-op on a proven flonum) apply
unchanged.
Sound subset only: :double, never :long — an untyped integer can be a bignum and
fx-ops would overflow/diverge from jolt's arbitrary precision. So an integer
caller leaves a param generic; an escaped fn (unknown callers) keeps :any.
run-numwp.ss gate: cross-fn :double propagation incl. through a flonum-returning
helper, the integer-caller and escape negatives, and the full run-passes path
emitting fl* + entry coercion. make test / shakesmoke green, selfhost holds, 0
new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
The devirtualized protocol call emitted find-protocol-method on the inferred
record tag, but a record can satisfy a protocol via an Object/host-tag default
rather than a direct impl — find-protocol-method on its own tag misses that,
while protocol-resolve walks to the default. So a record relying on
(extend-protocol P Object ...) resolved under ordinary dispatch but applied #f
under devirt and crashed. Closed-world opt builds only; the gate previously
covered just direct inline/extend-type impls so it shipped green.
Emit devirt-resolve, which tries the static tag and falls back to
protocol-resolve on a miss — same fast path, correct regardless of how the
record satisfies the protocol. Mirrors jrec-field-at falling back to jolt-get.
The receiver binds to one temp so it feeds the resolve and the application
without double-evaluating a side-effecting arg 0.
Also widen the whole-program fixpoint to :any on hitting the iteration cap: a
non-converged pre-fixpoint is more specific than the least fixpoint, so seeding
it would be unsound. Not reached in practice (~2 passes); a defensive floor.
run-devirt.ss gains an Object-default case. make test / shakesmoke green,
selfhost holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the inference types a keyword-lookup receiver as a record — it carries the
field-order :shape and :hint :struct from the whole-program fixpoint — the back
end reads the field by its declared slot via jrec-field-at instead of jolt-get.
That skips the jolt-get case-lambda, the dispatch fn, and the field-key
hashtable lookup, leaving a jrec? check + a static-index vector-ref.
jrec-field-at falls back to jolt-get when the receiver isn't the expected record
(a map downgraded by dissoc, or a value the inference mistyped), so it stays
correct if the static type is wrong. Only the no-default form takes the bare
path (a declared field is always present).
Sound only for non-nil records: a self-recursive param that can be nil (e.g.
binary-trees check-tree, whose untagged child is nil at leaves) types :any and
keeps jolt-get — the whole-program fixpoint demotes it. The target is non-nil
record params, like a Vec3 dot product (~5% there; boxed-flonum arithmetic
dominates the rest, a separate numeric lever).
run-fieldread.ss gate: emitted form uses jrec-field-at at the right slot and
matches jolt-get for each declared field; a non-field key and a default-arg form
keep the generic path. make test / shakesmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the inference proves a protocol call's receiver is one record type, the
back end resolves the impl by that static tag (find-protocol-method) instead of
routing through the protocol var -> jolt-invoke -> protocol-resolve, which
re-derives the tag and walks the type table. Same table lookup, minus the
var-deref, the rest-cons, and the receiver-type computation.
Fires only on a monomorphic site: a megamorphic receiver joins to :any and
carries no :devirt-type, so it keeps ordinary dispatch (the dispatch bench is
unaffected). The annotation comes from the whole-program fixpoint typing a
reduce/HOF element or a ctor return as a specific record.
Modest on the dispatch benchmarks (~6% on mono-dispatch) — float boxing in the
reduce accumulator dominates there, a separate numeric lever — but it removes
the dispatch overhead wherever a typed receiver is known.
run-devirt.ss gate: emitted form uses find-protocol-method, and evaluating it
matches ordinary dispatch for an inline impl, an extend-type impl, and the
non-devirt path. make test / shakesmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
Re-derive each app fn's param types from its call sites under --opt, so a
record type flows across fn boundaries: a ctor's return reaches a callee
param, and a typed vector's element reaches a HOF closure's param. The back
end can then bare-index field reads and devirtualize protocol calls at those
sites (it reads the resulting :hint/:devirt annotations; consuming them is
separate work).
This rebuilds the inter-procedural driver the Janet host had — the API
(infer-body/reinfer-def) survived the rehost but nothing drove it, and the
record-shapes/protocol-methods registries were empty stubs.
- records.ss: populate record-shapes (ctor key -> fields/tags/type, resolving
nested record field tags) and protocol-methods (method var -> [proto method])
registries at deftype/defprotocol load time; jolt.host accessors materialize
them.
- passes/types.clj: wp-infer! runs a closed-world fixpoint joining call-site
arg types into callee params; reinfer-def re-seeds each def at emit. Self-
recursive calls and fn-level recur are collected so a recursive fn's params
are constrained by its recursion, not just external callers — else a param
the recursion widens (e.g. binary-trees check-tree, whose untagged child can
be nil) would be unsoundly typed non-nil. A fn used in value position keeps
:any params (callers unknown). Megamorphic sites join to :any.
- build.ss: analyze all app forms and run the fixpoint before per-form emit.
- run-wp.ss: gate (cross-fn propagation, escape soundness, self-recursion).
make test / shakesmoke green, 0 new divergences, selfhost holds.
Co-authored-by: Yogthos <yogthos@gmail.com>
A loop test like (or (>= i cap) (> ... 4.0)) desugars to
(let* [g (>= i cap)] (if (truthy? g) g (> ... 4.0))) and the whole thing was
wrapped in jolt-truthy? because returns-scheme-bool? only looked at :const and
:invoke nodes, not the let*/if an or/and expands to. The wrapper defeats Chez's
branch inlining on the hot loop edge.
Make returns-scheme-bool? recursive over :if (both branches bool), :let (body
bool, tracking which bound locals hold a Scheme boolean), and :local (in that
set). or/and over bool-returning ops then read as Scheme booleans and the outer
wrapper drops. Still sound: eliding only when the value is provably #t/#f — a
jolt-nil is a truthy record in Chez, so a false positive would be a real bug, and
the recursion only proves bool-ness through ops already known to return one.
No bench regression; the win lands on hinted float loops where the branch, not
boxed arithmetic, is the cost.
Co-authored-by: Yogthos <yogthos@gmail.com>
defprotocol emitted one variadic (fn [this & rest] (protocol-dispatch P m this
(list->cseq rest))) per method, so every protocol call — even a no-extra-arg one
like (area s) — consed a rest list, wrapped it in a cseq, var-deref'd
protocol-dispatch, and jolt-invoke'd it (consing again). On mono-dispatch that was
2.07GB of allocation, ~65% of the benchmark.
Emit one fixed-arity clause per declared arglist instead. The 1/2/3-param arities
call positional protocol-dispatch{1,2,3}, which resolve the impl (by record tag,
reify method, or host-tag extension — factored into protocol-resolve) and apply it
directly; no rest-list, no seq round-trip. The dispatchN entry points are in the
native-op table so the shim calls bind straight to the records.ss procedures
rather than var-deref. 4+ params fall back to the variadic protocol-dispatch.
mono-dispatch 1.5s/2.07GB -> 0.69s/280MB; dispatch 26x -> 12.2x, mono-dispatch
111x -> 51x vs JVM. 5 new corpus rows pin multi-arity methods, host-type args,
and protocol-method-as-value against JVM Clojure.
Co-authored-by: Yogthos <yogthos@gmail.com>
Records were a jrec holding an alist of (kw . val) conses: ~113B/node, built
fresh per construction, field reads a list scan. Replace that with a shared
per-type descriptor (tag + field keywords + an eq?-keyed keyword->index table)
plus a flat per-instance value vector and an extension map for any non-field
keys assoc'd on (jolt-nil when there are none). Construction now allocates one
vector instead of a cons chain and a field read is an index lookup. binary-trees
construction allocation drops 2.085GB -> 1.19GB.
That alone barely moved binary-trees wall-time: profiling showed the read loop,
not allocation, dominates, and the read loop's own allocation came from (nil? l)
lowering to (jolt-invoke (var-deref "clojure.core" "nil?") l), which conses its
args every call. Add nil?/some? to the backend native-op table so they inline to
jolt-nil?/jolt-some? (and drop the truthy wrapper, like the other predicates).
check-tree's read loop goes from 1.476GB allocated to zero; binary-trees 18.9x
-> 9.7x vs JVM. The remaining gap is the field-read dispatch chain (jolt-c3mw).
Two JVM divergences fixed along the way, both certified:
- dissoc of a declared field downgrades a record to a plain map (was kept as a
record); an extension key still drops cleanly.
- map->R keeps extension keys (was dropping anything outside the declared basis).
16 new corpus rows pin assoc/dissoc/count/keys/seq/=/hash/extension-field
behavior against JVM Clojure.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Make the benchmark harness build optimized binaries on Chez
bench/run.sh was Janet-era: it invoked a 'jolt' binary and set
JOLT_DIRECT_LINK/JOLT_WHOLE_PROGRAM, none of which exist on Chez, where
'joltc run -m' runs fully unoptimized (direct-link and inline default off). So
the suite was measuring jolt's unoptimized path.
run.sh now compiles each benchmark to an optimized AOT binary (joltc build
--direct-link --opt) and times it against JVM Clojure on the same portable
source, auto-detecting the Chez kernel dev files like build-smoke.sh. Adds
bench/deps.edn so joltc resolves the namespaces, NO_JVM to skip the reference.
mandelbrot.clj dropped its jolt.png require so the JVM reference can run it; the
picture demo moved to mandelbrot_png.clj (jolt-only). README scorecard refreshed
with current Chez numbers and the two-regime read (compute ~8-10x substrate floor;
dispatch/alloc ~120-330x architectural gaps the passes don't touch). Stale
'jolt -m' header lines point at bench/run.sh.
* Emit direct self-calls for named-fn self-recursion
A self-recursive call to a named fn compiled to (jolt-invoke fib ...) instead of
a direct (fib ...): emit-invoke handled a :local callee only when it was NOT a
known proc, so a :local that IS in *known-procs* (the letrec-bound self-name) fell
through to the :else jolt-invoke branch. Now a :local known proc emits a direct
Scheme call — no jolt-invoke, no per-call arg-list consing; case-lambda handles
arity.
fib 30: 63.3ms -> 4.7ms (faster than JVM Clojure's 7.1ms; was 9x slower). The win
is on every self-recursive non-loop fn, including the compiler's own. No semantic
change — selfhost holds, make test green, shakesmoke/buildsmoke byte-identical.
Re-mint (backend is seed). Corpus rows pin self-recursion across fixed/multi/
variadic arities.
* Intern no-ns keywords without per-call allocation
(keyword #f name) built a fresh combined-key string (string-append) on every
call just to do the intern-table lookup — ~80 bytes of garbage per (:kw x), map
literal, keyword arg, etc. A no-ns keyword now interns in a table keyed by the
name string directly, so a lookup of an already-interned keyword is one
hashtable-ref with no allocation. The ns table keeps the combined key; both share
the keyword-t khash (equal-hash of the combined key) so hash values are unchanged.
Small time win on its own (the field-read dispatch dominates hot record code —
see jolt-unx4) but removes per-call keyword allocation everywhere. Runtime .ss,
no re-mint; identity/=/hash unchanged, make test green.
* Fast record field reads: single eq? scan, skip the get-arm walk
(:field rec) / (get rec :field) lowers to (jolt-get rec kw), which walked the
get-arm list to reach the jrec arm, then did jrec-has? + jrec-lookup — TWO linear
scans, each comparing keys through the generic jolt=2 equality dispatcher. Field
keys are interned keywords, so:
- jrec-key=? compares a keyword query by eq? (jolt=2 only for non-keyword keys),
- jrec-ref does ONE scan (vs has?+lookup) and runs a deftype's ILookup valAt only
when the field is genuinely absent (present-nil still returns nil, not default),
- jolt-get-dispatch checks jrec? first, skipping the get-arm walk for the hottest
get target. jrec-lookup/jrec-has? (used by =, contains?, etc.) get the fast
compare too.
binary-trees 135x->18.9x, dispatch 121x->26.4x, mono-dispatch 327x->108x vs JVM.
Runtime .ss (collections.ss + records.ss), no re-mint; make test + shakesmoke +
buildsmoke green, record get/assoc/keys/=/count semantics unchanged.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
A type-aware audit (~190 collection expressions vs reference Clojure) found four
divergences the corpus missed — value-equality (= [0 1] '(0 1)) hides type and
laziness differences. Fixed, with type-predicate + over-infinite corpus rows that
pin them.
- partition-all [n coll] built vector chunks; JVM chunks are seqs. (The [n step
coll] arity was already correct, as is the partition-all transducer, whose
chunks are vectors in JVM too.) Now builds seq chunks.
- replace always returned a vector (mapv) and was eager; JVM is type-preserving —
a vector maps to a vector, any other seqable to a lazy seq.
- sequence eagerly realized its source (into-xform), so (first (sequence (map inc)
(range))) hung. Rewrote as a transformer iterator: pull one input at a time,
buffer the step outputs, emit lazily, run the completion to flush a stateful
xform. eduction builds on it (lazy, no longer an eager vector).
- mapcat and (apply concat coll-of-colls) hung over an infinite source because
jolt-apply seq->lists the trailing arg and mapcat seq->lists the map result.
Added lazy-concat-seq (lazily flatten a seq of colls); mapcat uses it directly,
and apply special-cases concat (its result is lazy) to route through it.
Docs: a cross-cutting return-type + laziness contract in docs/spec/09-core-library;
SPEC.md notes that = masks type/laziness so they need predicate / over-infinite
rows. EBNF is reader syntax only — unaffected.
Seed change (partition-all/replace/eduction are clojure.core overlay) -> re-mint;
selfhost holds. make test + shakesmoke + buildsmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Reader records source line/column on list forms
The reader stamps 1-based :line/:column metadata on every list form (plus
:file when load-jolt-file is reading a file), and jolt.host/form-position
reads it back so the analyzer's :pos scaffold finally gets real data. A
left-to-right cursor counts newlines over the delta between successive forms,
so it stays O(n). Vector/map/set literals are untouched (their metadata is a
runtime value the analyzer would have to wrap in with-meta); empty () can't
carry meta. ^meta now merges onto the position keys instead of clobbering them.
Re-mint is byte-identical (the backend doesn't emit :pos), so this is a pure
scaffold for the error-location work that follows.
* Report source location on uncaught errors
Each top-level form records its source position (thread-local) before it
compiles+evals, and cli.ss jolt-report-uncaught appends 'at file:line:col'
when an error propagates out. Covers joltc -e, joltc run <file>, and
load-string — every interpreted path. Top-level granularity, one set per
form; deeper frames come from the Phase 2 frame walk.
Runtime .ss only, no re-mint.
* Clojure stack traces via source registry + native frame walk
A direct-link build emits (jolt-register-source! short-name ns name file line)
once per fn def — at definition time, so zero per-call cost. On an uncaught
error the reporter walks Chez's native continuation frames (jolt-throw captures
the live continuation via call/cc; host conditions carry their own
&continuation), maps each frame's procedure name through the registry, and
prints a Clojure backtrace 'ns/name (file:line)'. Wired into both the cli and a
built binary's launcher.
Frames are keyed by the short munged fn name Chez actually reports (emit-fn's
letrec self-binding), not jv$ns$name; a cross-namespace collision degrades to
the bare frame name rather than a wrong attribution. The analyzer carries the
original form's position through defn macroexpansion onto the def node.
Calling a non-fn now throws a catchable ClassCastException (via jolt-throw)
naming the operator, instead of a raw Chez error.
Caveats (documented in source-registry.ss): names map only in direct-link/AOT
closed-world builds — the open-world -e/repl/run path falls back to the
top-level location; and pervasive TCO erases tail-call frames, so a mapped
trace shows only the non-tail spine. JOLT_DEBUG_FRAMES dumps raw frame names.
Re-mint (analyzer + backend); prelude byte-identical (direct-link off during
mint). Corpus rows certified, build-smoke asserts the trace.
* Propagate source position through macroexpansion
hc-expand-1 now carries the macro call form's :line/:column onto the top of a
list expansion that has none of its own (merged under any meta the macro set),
so errors and stack traces in macro-generated code point at the call site —
Clojure parity. The analyze recursion re-expands inner macros, so each level's
top form picks it up, matching the reference compiler. (meta (macroexpand-1
'(when x y))) now reports the call-site line.
A direct-link fn defined through a user macro (build-app's defguarded) registers
with a real line, so build-smoke's trace assertion covers macro-defined fns.
Runtime .ss (host-contract.ss) — no re-mint; selfhost holds.
Phase 3's optional items are deferred: :line-in-ex-data has no clean consumer
(it would pollute ex-data, break = and printing, and positions already surface
via the trace + top-level location), and Chez source-object emission is a large
backend change the jv$-name registry already sidesteps.
* Review fixes: registration key, thread-locals, debug flag timing
- Register a fn under the name Chez actually reports for its frame, not the def
name: a named fn literal whose name differs from the def (def foo (fn bar …))
is framed as 'bar', and an anonymous fn def (def foo (fn …)) as jv$ns$foo.
Both previously registered under the def name and so never appeared in traces.
- rdr-source-file / rdr-pos-cursor are thread parameters, so concurrent compiles
(futures, core.async) don't clobber each other's file/line attribution.
- Read JOLT_DEBUG_FRAMES at call time: a built binary evaluates top-level forms
at heap-build time, where a load-time getenv is always unset.
Re-mint (backend + reader); prelude byte-identical, selfhost holds.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Review found (< 1M 2M) worked but (min 1M 2M) threw — incoherent. Wire min/max
the same way as the other ops: value-position jolt-min/jolt-max shims (new in
seq.ss, added to core-value-procs) and call-position via bd-spec/bd-ops ->
jbd-min/jbd-max.
min/max return the original operand by value, not a coerced copy, matching
Clojure: (min 1M 2.0) -> 1M, (max 1M 2.0) -> 2.0, (min 1.50M 2M) -> 1.50M; a tie
keeps the second operand ((max 1.5M 1.50M) -> 1.50M). bigdec mixed with a flonum
in call position stays in the documented :any/contagion gap (value position
handles it). Re-mint; 6 more JVM-certified rows.
A direct (+ 1.5M 2.5M) emits a raw Chez + that rejects the bigdec record. Rather
than guard every arithmetic call site (measured 2-4x on unhinted fixnum loops),
let the analyzer dispatch where it can prove the type.
jolt.passes.numeric seeds a :bigdec kind from the M-literal and flows it through
let/loop/if like the existing :double/:long kinds; an arithmetic/comparison invoke
whose operands are all bigdec (integer literals allowed) gets :num-kind :bigdec.
The back end (bd-ops + emit-numeric) lowers those to the bigdec.ss engine
(jbd-add/-sub/-mul/-div, jbd-lt?/…, jbd-zero?/-pos?/-neg?, jbd-quot/-rem).
Zero cost on non-bigdec code: with no bigdec literals present the kind never
arises, so emission is byte-identical — the re-mint leaves prelude.ss unchanged,
only image.ss (the compiler) moves. Gaps (filed): a bigdec mixed with a flonum in
call position, and a bigdec the analyzer types :any, still hit the raw op and
throw; use value position or a literal-typed let.
Re-mint (numeric/backend are seed sources). 16 JVM-certified corpus rows.
