The transient assoc! accepted an odd key/val count and silently assigned
nil to the dangling key — non-Clojure, and inconsistent with the seed's
own plain assoc (which throws) and Clojure's assoc!. Make it throw.
Updates the 4 'assoc! odd args' transient spec rows to 3 :throws + 1
even-args positive, and regenerates corpus.edn. The Chez host already
threw on these, so this only realigns the corpus contract.
host/chez/dynamic-vars.ss binds the two seed-native dynamic vars that aren't
emitted into the prelude: *clojure-version* (the {:major 1 :minor 11 ...} map) and
*unchecked-math* (false). Removes their two run-corpus-prelude allowlist entries
(now 11, all passing).
*ns* is deferred to jolt-b4kl: it needs a namespace value that is not a map yet
answers (get ns :name) for the overlay ns-name, plus str/find-ns support.
Parity 1530 -> 1532/2497, 0 new divergences. emit-test 305/305.
host/chez/multimethods.ss implements the multimethod runtime: defmulti/defmethod
expand to defmulti-setup/defmethod-setup calls (+ get-method/methods/
remove-method/prefer-method/prefers). A jolt-multifn record carries its dispatch
fn and a jolt=-keyed method table; jolt-invoke dispatches it (exact match, then
isa?/hierarchy with prefer-method, then :default), reusing the overlay's
isa?/derive/make-hierarchy. The multifn's ns comes from a runtime chez-current-ns
(default user; the prelude load sets clojure.core for print-method/print-dup).
Two emit-side changes were needed:
- late-bind (:late-bind-unresolved? ctx flag, default OFF): defmulti expands to a
bare-symbol setup call, so the analyzer doesn't intern the name and a forward
reference '(area ...)' after '(defmulti area ...)' in one form was 'Unable to
resolve symbol'. The strict compiler punts these to the interpreter; the Chez
back end has none, so the flag lowers an unresolved symbol to a var-ref against
the compile ns (open-world -e semantics). Set only by the Chez make-ctx /
jolt-chez; the main compiler keeps strict resolution (host_iface late-bind?
defaults nil).
- a :var call head now routes through jolt-invoke, since a late-bound var can hold
a multifn (or keyword/coll IFn), not just a procedure. Transparent for
procedures; the hot self-recursive call is a :local known-proc, stays direct.
Class-based dispatch ((class x)/String) deferred (needs deftype/class subsystem).
Parity 1506 -> 1530/2497, 0 new divergences. emit-test 302/302. Full janet gate
green (the analyzer flag is off there; suite flakiness under parallel load only).
jolt-y1zq tail:
- 0-arg (conj) -> [] and 0-arg (conj!) -> a fresh transient vector
- nth sees through a transient (like get/count/contains?)
- irregex \p{...}/\P{...} property classes translate to the seed's ASCII char
classes (regex.ss): \p{L} -> [a-zA-Z] + non-ASCII codepoints (the seed counts
UTF-8 high bytes as letters), \p{N} -> [0-9], \p{Ps} -> [([{], etc. The
translator tracks [...] nesting so a \p{} inside a class emits its content, not
a nested class.
Two pre-existing bugs found and filed (tracked, not replicated):
- jolt-x0os: the Chez emitter mangles non-ASCII string literals into invalid Chez
hex escapes (so p{L} utf-8 crashes on the input string, not the regex).
- jolt-ea9k: the seed's transient assoc! accepts odd args and assigns nil to the
trailing key (non-Clojure; plain assoc throws). The Chez host throws
(Clojure-correct); the 4 spec rows encoding the leniency are flagged in
transients-spec.janet pending the seed fix.
Parity 1493 -> 1506/2497, 0 new divergences. emit-test 291/291.
The 1-arg map/filter/remove/take/drop/take-while/drop-while/mapcat now return a
transducer (fn [rf] rf'), and into gets a 3-arg (into to xform from). This was
the 'cdr () is not a pair' / 'incorrect number of arguments' crash bucket: the
emitter lowers (map f) and 3-arg into at an arity the native-op gate rejects, so
they fall to the value-position path and hit the bare jolt-map/jolt-into
procedure at the wrong arity. The fix is RT-side — case-lambda those procedures
plus jolt-into.
td-* factories ported from the seed (core_coll.janet); a reduced step stops the
fold via reduce-seq's existing short-circuit (inc 3n). transduce/comp/completing
are overlay and compose over these unchanged.
Parity 1467 -> 1493/2497, 0 new divergences. emit-test 278/278.
The dominant prelude-parity crash bucket was 'apply non-procedure jolt-nil':
core fns calling seed-native seq fns (core_coll.janet) that have no Chez RT
shim, so var-deref returns jolt-nil. A static scan of the assembled prelude
turned up 52 referenced-but-undefined clojure.core names.
host/chez/natives-seq.ss shims the safe seq fns over the existing seq layer:
mapcat, take-while, drop-while, partition (collection arities only — the 1-arg
transducer forms are jolt-kxsr), and sort (compare default; a comparator may
return a 3-way number or a boolean less-than). reduced/reduced? is a jolt-reduced
record in seq.ss that reduce short-circuits on and deref unwraps, so unreduced
works. identical? = jolt= (the seed's definition).
Deferred list?: a Chez lazy seq and a list are both cseq, so it can't be told
apart without a distinct list type — a real divergence risk.
Parity 1407 -> 1467/2497, 0 new divergences. emit-test 263/263.
##Inf/##-Inf/##NaN were emitted to bare inf/-inf/nan, which are unbound symbols in
Chez. emit-const now lowers them to +inf.0/-inf.0/+nan.0. The -e/element printer
renders them inf/-inf/nan (Chez's number->string gives +inf.0), and str renders the
long Clojure forms Infinity/-Infinity/NaN. assoc! is now variadic ((assoc! t k v
& kvs)) like Clojure.
Prelude parity 1382 -> 1407/2497, 0 new divergences. str of inf INSIDE a collection
still wants the long form (needs the Phase-2 recursive str renderer), so
[inf inside coll] is allowlisted. Transducer arities and the cdr-on-()/\p{} regex
gaps are split out to jolt-kxsr/jolt-y1zq.
transient/persistent!/conj!/assoc!/dissoc!/disj!/pop! as copy-on-write over the
persistent collections (host/chez/transients.ss) — each op rebuilds the persistent
coll (no in-place perf) but the semantics match, so into/frequencies/group-by work.
Adds persistent disj over pset-disj. get/count/contains? are redefined to see
through a transient (frequencies and group-by both do (get tm k) on a transient
map); vector? on a transient vector is false, which group-by relies on.
Prelude parity 1326 -> 1382/2497, 0 new divergences. emit-test exercises the direct
transient ops via run-prelude and the overlay users (frequencies/group-by/into)
end-to-end through the bin/jolt-chez -e binary.
str/subs/vec/keyword/symbol/compare/int/double/gensym — host-coupled seed natives
the overlay assumes, now def-var!'d into clojure.core via host/chez/converters.ss
(loaded last, str reuses jolt-pr-str). Semantics match the seed: str-render-one
for str (nil->"", bare chars, regex source), the 3-way core-compare port, int
truncates. The symbol no-ns sentinel is #f to match emit's quoted-symbol lowering
(jolt-symbol #f "x"), so (= 'x (symbol "x")) holds — jolt= compares ns with strict
equal?, and jolt-nil vs #f would otherwise diverge.
Prelude parity 1220 -> 1326/2497, 0 new divergences. Floor raised to 1326; three
newly-reached *ns*/var-rendering cases added to the allowlist (Phase 2). run-prelude
and the per-case program file are now PID-unique so concurrent chez runs don't read
each other's half-written files.
Emit every non-macro clojure.core form through the live analyzer -> Chez emit
pipeline as a def-var! prelude (prelude mode, tier dependency order), load it
before a user expression, and you get an -e-capable jolt-chez: analysis on Janet,
execution on Chez. driver/emit-core-prelude assembles it (each form behind a
silent load guard so the Phase-2 multimethod forms don't break the rest);
bin/jolt-chez is the -e CLI, caching the prelude on disk keyed by source hash.
run-corpus-prelude.janet is the full parity gate this opens, the prelude-backed
sibling of run-corpus-chez. First baseline: 1220/2497 evaluated cases pass, 0 new
divergences (10 allowlisted: dynamic vars, class names, eval-order — deferred
Phase 2). The rest is the punch-list: ~360 emit-fail (real host interop, out of
the analyzer subset) and ~900 runtime crashes, mostly core fns calling
host-coupled seed natives with no Chez shim yet (str/format/vec, transients).
Follow-ups jolt-t6cr/kl2l/q3w8/9ls5.
Two shims landed to get the prelude to load and run. atoms.ss: atom/deref/swap!/
reset! (+ the compare/vals kernel) — needed at load time for
global-hierarchy = (atom (make-hierarchy)). predicates.ss: the type predicates +
name/namespace/boolean the overlay assumes are seed natives, matching the seed's
strict semantics. post-prelude.ss re-asserts char?/atom? after the prelude: the
overlay implements those by reading :jolt/type, which is false for Chez-native
chars/atoms, so its def-var! would clobber the correct native versions.
Per-case Scheme files are PID-unique so a foreground -e never reads a half-written
file while the gate runs.
Closes the last clojure.core prelude emit gap (parse-uuid): the whole
non-macro core now lowers to Scheme (prelude reach 355/355).
A #"..." literal analyzes to a :regex IR node. The Chez back end emits
a jolt-regex value over irregex (Alex Shinn, BSD), vendored as the
vendor/irregex submodule -- a portable Scheme regex with PCRE/Java-style
string patterns and first-class Chez support. host/chez/regex.ss wraps
jolt's re-* surface over it: irregex-match -> re-matches (anchored),
irregex-search -> re-find, groups as Clojure [whole g1 ...] vectors,
re-seq as a jolt seq. re-pattern/re-matches/re-find/re-seq/regex? are
def-var!'d into clojure.core so prelude / -e code resolves them.
They stay OUT of the subset native-ops on purpose: irregex's
Unicode/property-class semantics differ from the seed's byte-PEG
approximation, so keeping them prelude-only avoids dragging
engine-difference divergences into the subset-parity corpus. The Janet
back end punts :regex to the interpreter (the seed compiles #"..." to a
Janet PEG), so the main language is unchanged.
Only two adaptations for Chez's top level: a cond-expand shim (Chez's is
library-only) and a normalizing error wrapper (silences irregex's 1-arg
error warnings). rt.ss load is ~0.18s.
emit-test 131/131 (regex literal + re-* parity vs the CLI oracle);
prelude reach 355/355; Chez subset 672/672, 0 divergences; full gate
green.
(.method target arg*) now analyzes to a :host-call IR node instead of
punting at analyze. The Chez back end lowers it to a jolt-host-call
dispatch for the methods the RT shims (.write -> port display,
.isDirectory -> file-directory?, .listFiles -> directory-list); any
other method stays out of subset (clean emit-time reject, so it can't
read as a compiled-but-broken corpus divergence). The Janet back end
punts ALL :host-call to the interpreter, same shape as letfn: compiles
on Chez, interprets on Janet, zero change to the main language.
Closes the io tier's print-method defmethods and file-seq: prelude emit
reach 348 -> 354/355 (50-io 20/20). The one remaining gap is the regex
literal in parse-uuid (needs a regex engine on Chez; deferred).
emit-test 122/122; Chez subset 672/672, 0 divergences; full gate green.
Closes the last two non-host-interop prelude emit gaps.
letfn now analyzes to a :let node flagged :letrec — the binding fns are bound
into the env together before any spec is analyzed, so siblings and self resolve.
The Chez back end lowers it to letrec*; the Janet back end punts it at emit
(its sequential let* can't express the mutual recursion — same interpreter
fallback as before, just decided at emit-ir instead of analyze).
(def x) with no init (declare) analyzes to a :def with :no-init instead of
punting. Chez reserves the var cell via declare-var! (which doesn't clobber an
existing root — (do (def x 7) (def x) x) => 7); the Janet back end still punts
to the interpreter, which interns a genuinely-unbound var.
fallback-zero-test now checks emit-ir too, not just analyze-form, so the real
compile-vs-interpret decision is what it asserts (letfn/def-no-init analyze but
the Janet back end punts them). letfn stays in must-punt with an updated note.
Prelude emit reach 342 -> 348/355 (40-lazy now 13/13); Chez subset 664 -> 672,
0 divergences; emit-test 110 -> 117. Full gate green.
Emit a :quote node by reconstructing the raw reader form as RT constructor
calls: symbol -> jolt-symbol, list (array) -> jolt-list, vector (tuple) ->
jolt-vector, map -> jolt-hash-map, set -> jolt-hash-set, scalars via emit-const.
The runtime value of a quote is just that literal data (the interpreter returns
the reader form verbatim).
Quote exposed a latent seq.ss bug: empty map/filter results were jolt-nil, but
Clojure's (map f []) is an empty seq, so (= () (map f [])) must be true. Return
jolt-empty-list (which seqs back to nil, so it's still a valid lazy-tail
terminator) instead — matching jolt-take/drop/rest/list.
Prelude emit reach 334 -> 342/355. Subset probe 632 -> 664/664 compiled, 0
divergences (quote + the seq fix pull 32 corpus cases into the subset). emit-test
110/110 (added 16 quote cases). corpus.edn regenerated (the 3 malformed-catch
spec rows). Full gate green.
jolt's catch is (catch class binding body*); the binding (3rd element) must be
a symbol. Neither the analyzer nor the interpreter validated it, so a non-symbol
binding crashed with an internal Janet error (expected integer key for array...)
and, in the interpreter, a malformed clause whose body never threw was silently
swallowed (returned the try value). Clojure rejects a non-class/non-symbol catch
clause; match that with an up-front error in analyze-try and eval-try.
Surfaced building the Chez try/throw emit. Regression rows in exceptions-spec
(runs x3 modes) plus a unit test asserting the clean message in interpret and
compile. jolt-kg6p.
Emit :throw as jolt-throw (Scheme raise of the raw jolt value, matching the
Janet compiled back end's (error v) — no envelope, so catch binds it directly).
Emit :try as guard (catch; the class is dropped in the IR, so it's catch-all)
plus dynamic-wind for finally. ex-info is a native-op building the tagged jolt
map {:jolt/type :jolt/ex-info :message :data :cause}, so the ex-data/ex-message/
ex-cause tier fns read it over jolt-get for free.
Prelude emit reach 303 -> 334/355 (:throw and :try gaps close). Subset probe
619 -> 632/632 compiled, 0 divergences (throw/try/ex-info pull 13 corpus cases
into the subset). emit-test 94/94 (added 11 throw/try/ex-info cases + uncaught
exits non-zero). Full gate green.
Add a prelude emit mode to host/chez/emit.janet: when emitting clojure.core
itself (not user -e), a non-native clojure.core ref lowers to a runtime
var-deref instead of being rejected as out-of-subset, so core fns chain
through each other. Default (subset) mode is unchanged — the corpus probe
still rejects unimplemented core refs for a clean signal.
core-prelude-probe.janet walks the tiers through the live analyzer->emit
pipeline and catalogs reach + gaps (macros skipped; analyze-time only).
Baseline: 303/355 non-macro core forms emit. Remaining gaps are a tight
punch-list for the next increments: :throw (29), :quote (8), :try (2), Java
host interop (6), letfn (4), declare (2). Probe has a regression floor.
emit-test 83/83 (added prelude-mode lowering assertions); subset probe
619/619 unchanged; full gate green.
emit-fn lowered multi-arity fns to a Scheme case-lambda and variadic fns to
a rest-arg lambda; the Scheme rest list is coerced to a jolt seq (nil when
empty, via list->cseq), and the named-let wrapper runs that coercion only on
first entry so recur carries the seq directly. Single fixed arity keeps the
plain-lambda fast path (fib untouched).
Also fixes a latent leak in the module-global known-procs: a throw mid-emit
(uncompilable body) left the fn's name registered, so a later corpus case
binding the same name to a keyword emitted a direct call to a non-procedure.
The cleanup now runs on the error path too. Only surfaced once the new arity
support let +24 cases compile further before hitting an uncompilable fn.
Gate: emit-test 81/81, subset probe 619/619 compiled (was 595), 0 divergences,
2036/2655 out of subset; full run-tests green (125 files).
Brings up the seq layer on the Chez runtime. host/chez/seq.ss adds one
lazy-capable node (cseq) that models Clojure's list, cons, and lazy seq -
all print as (...), all sequential-= to each other and to vectors. seq
coerces any seqable (vector/map/set/string/list/seq/nil) to a cseq or nil;
the empty seq is a distinct value printing () (rest of a 1-elem coll is ()
not nil, seq of empty is nil). Leaf ops: first/rest/next/seq/cons/list,
reverse/last, map/filter/remove/reduce/into, range/take/drop/concat/apply,
keys/vals, plus nth/peek/pop extended over seqs. map/filter/reduce apply
their fn arg through jolt-invoke, so a procedure, keyword, or collection all
work as the fn.
Dynamic IFn dispatch: a keyword/vector/coll held in a local (let binding or
fn param) and called as a fn now routes through the jolt-invoke fallback
(procedure? -> apply; keyword/coll -> lookup). The emitter only routes a
:local callee that isn't a known procedure - a named fn's self-recursion
name stays a direct call, so the fib hot path is untouched. Closes the 3
ex-known IFn divergences.
emit.janet: seq/pred ops added to native-ops with arity gates; value-position
clojure.core refs resolve to the RT procedure (native-ops names one for each),
with +/-/*// routed to flonum-coercing wrappers so higher-order arithmetic
((reduce + [])) keeps the all-double model. values.ss: cross-type sequential
=/hash so a vector and a list of the same elements are jolt= and hash alike.
rt.ss: printer learns seqs; top-level nil prints as the empty string (jolt -e
str-style). Fixed latent bug: (conj nil ...) now builds a list, not a vector.
Gates: emit-test 69/69 (fib/mandelbrot/collections/seq/IFn parity vs the jolt
oracle, fib(30) ~24ms unchanged). Subset probe 433/436 -> 595/595 compiled,
0 divergences (was 3 known), 2060/2655 out of subset. Full run-tests green
(125 files, conformance + suites included).
Broaden the Scheme back end past the numeric/functional subset to vectors,
maps, and sets. host/chez/collections.ss adds a copy-on-write persistent
vector and a bitmap HAMT (the structure 0c measured self-hostable) backing
both maps and sets, keyed by jolt-hash and compared by jolt=. emit.janet
emits :vector/:map/:set literals to the rt constructors and lowers the leaf
ops (conj/get/nth/count/assoc/dissoc/contains?/empty?/peek/pop) via the
native-ops path, with a per-op arity gate.
Also: keyword/map literals in fn position lower to jolt-get ((:k m), ({:k v} k));
arity-1 comparisons emit the vacuous jolt truth (Scheme < rejects a non-number
even at arity 1); count returns a flonum and vector indices coerce from flonum,
both consequences of the all-double number model; values.ss = / hash and the
rt printer learn collections (maps/sets render in HAMT order, so the probe
compares unordered values via =, not printed form).
Subset parity 182 -> 433/436 compiled cases (2219/2655 out of subset), 0 new
divergences. The 3 known divergences are dynamic IFn dispatch (a keyword/vector
held in a local, called as a fn) — deferred to the IFn/protocol increment and
allowlisted in the probe. emit-test 31/31, full run-tests green (125 files).
Wire the real pipeline end to end: host/chez/driver.janet boots a compile-mode
jolt ctx, runs the EXISTING Janet-hosted analyzer on actual Clojure source to
real IR, feeds it to the Scheme emitter, and runs the result on Chez. Analysis
stays on Janet (the analyzer ports to Chez in Phase 2); execution is on Chez.
emit.janet now consumes live IR (pv/phm-normalized like the Janet backend) and
covers what the analyzer actually emits, not the hand-built inc-1 shapes:
- core ops arrive as :var clojure.core/+ etc., not :rt — lowered to native
Scheme via a native-ops table (mirrors backend.janet's), `=` to jolt=.
- var cells (host/chez/rt.ss): :def -> def-var!, :var -> var-deref. Late binding
so cross-var calls (run -> count-point) and the entry crossing resolve at use.
- named fns (defn / fn self-name) bind via letrec so self-recursion resolves.
- unsupported stdlib/host refs (no core on Chez yet) are rejected at EMIT time
(clean out-of-subset signal) instead of deref'ing to nil and failing at runtime.
Number model: jolt is all-doubles (no ratios; (/ 1 2) is 0.5), so literals emit
as flonums — matches the Janet host and keeps Chez out of exploding exact
rationals (mandelbrot). jolt-num->string prints integer-valued without ".0".
Two real bugs found via the corpus probe and fixed (regression rows added):
- loop bound in parallel (Scheme named-let) but Clojure loop is sequential — a
later init must see earlier bindings; wrap a let* around the loop.
- #(...) shorthand gensyms params with a trailing `#`, invalid in Scheme — munge
it to `_`.
Gate: test/chez/emit-test.janet runs the real analyzer -> Chez for (+ 1 2),
fib(30)=832040, mandelbrot run(40), and the two regressions, parity-checked
against the Janet oracle (6/6). First parity number via the new subset probe
(test/chez/run-corpus-chez.janet, JOLT_CHEZ_CORPUS=1): 182/182 compiled corpus
cases pass, 0 divergences; 2473/2655 out of subset pending core on Chez. Full
jpm/run-tests gate green (125 files). Chez tests skip cleanly without `chez`.
Perf note (unchanged plan): emitted fib(30) ~23ms vs hand-Scheme ~5ms — the
jolt-truthy? wrapper (~3x) plus flonum (not fixnum) arithmetic, both Phase-4
type-specialization levers.
New back end half: host/chez/emit.janet consumes the host-neutral jolt IR
(ir.clj shapes) and emits Scheme, reusing the existing front-end (Option-2
backend swap). Covers the pure-functional subset: const/local/var/rt/if/do/let/
fn/invoke/def/loop/recur. Tested by hand-built IR run on Chez: (+ 1 2)=3,
fib(30)=832040, loop/recur sum=15 (4/4).
Finding: correct emit wraps every if-test in jolt-truthy?, costing ~3x on fib
(15.8ms vs hand-Scheme 5ms). Eliding the wrapper for known-boolean tests
recovers the ceiling (Phase-4 type-driven opt).
Remaining Phase 1: wire the live analyzer, var-cell late binding, RT module,
broader op coverage for mandelbrot.
0c: persistent HAMT on Chez is ~41x faster than Janet's HAMT on the collections
map-churn (258.6 -> 6.3 ms), ~15x off mutable-native (inherent persistence cost).
Decision: self-host the persistent collections in Clojure; substrate is not the
bottleneck. See docs/chez-phase0-results.md.
0a hardening: NUL-separated keyword intern key (no ns/name collision), non-finite
-safe jolt-hash. 37/37.
0a (host/chez/values.ss): Jolt value model on Chez — nil sentinel distinct from
#f/'(), interned keywords, ns+meta symbols, exactness-aware = and consistent
hash. Chez numeric tower gives ratios/bignums free. 33/33 tests.
0b (test/chez/): extract test/spec/*.janet defspec tables into corpus.edn (2655
cases, valid as both EDN and Janet data), and a runner that drives ANY jolt
binary via the CLI boundary with per-case subprocess isolation. Pluggable target
(JOLT_BIN) so the same corpus gates every host. Baseline vs Janet build/jolt:
2641/2655, 14 known CLI divergences allowlisted; gate fails only on NEW ones.
Generic language fixes so a host-shim library (jolt-lang/http-client) can carry
clj-http-lite without baking any HTTP/native code into core:
- reader: accept #^ (deprecated metadata reader macro) as ^
- (str pattern) returns the raw regex source, not #"..." — pattern composition
via (re-pattern (str p ...)) now works
- into/conj onto a map merges map items (was (into {} [{:a 1}]) -> {nil nil})
- a Var is callable as its current value (e.g. (wrap #'f) threading a var client)
- core-class reads a :class field off a thrown table, so libraries can throw
typed exceptions whose (class e) is a JVM class name
- compiled catch unwraps the interpreter's :jolt/exception envelope (__unwrap-ex),
so a typed throw keeps its class/message across the interpret/compile boundary
- slurp accepts a byte-array; io/copy is generic over :jolt/input-stream /
:jolt/output-stream stream markers
- instance? gains a registry (__register-instance-check!) for library shim types
- new clojure.test namespace (deftest/is/are/testing/use-fixtures/run-tests with
class-aware thrown?/thrown-with-msg?)
Spec: test/spec/clojure-interop-fixes-spec.janet.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Fix four bugs surfaced by the commonmark-app example
- regex: bounded quantifiers {n,m} no longer expand exponentially. The
desugaring inlined the continuation per optional level, doubling it each
time, so {0,61} built a PEG the compiler expanded to ~2^61 nodes and hung.
Each level is now its own grammar rule referenced by name (jolt-3xur).
- strings are a seqable of chars for vec/set/into, matching seq. They went
through realize-for-iteration, which had no string case, so into/set got
raw bytes (code points) and set threw; vec used string/from-bytes (1-char
strings). realize-for-iteration now maps make-char over the bytes, and
core-vec matches (jolt-dl4s).
- clojure.string/split takes Java Pattern.split limit semantics: negative
keeps trailing empties, 0/omitted drops them (a no-match result stays
[input]), positive caps the part count with the remainder unsplit. It used
to delegate the limit to take, so a negative limit returned [] and the
2-arg form never trimmed trailing empties (jolt-ik3a).
- System/exit is registered (maps to os/exit), so (System/exit n) works
(jolt-w2wf).
* regex: single-digit backreferences \1..\9 (jolt-xtss)
\1..\9 now match the text captured by the corresponding group, so
patterns like ([-*_])\1\1 or (\w+) \1 work. The parser records which
groups are referenced; a referenced group additionally captures its text
under a tag and the backref compiles to a PEG (backmatch). Only referenced
groups change — they match possessively (the CPS-over-possessive-PEG engine
can't backtrack into a tagged capture), so backtracking back into a
backreferenced group isn't supported (rare). Unreferenced groups keep full
backtracking and position-based result capture unchanged.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
canon-key canonicalizes a collection key by re-keying a native Janet
table by the canonical form of each element/entry. canon-key returns nil
for nil, and a Janet struct can't hold a nil key or value, so a nil set
member / nil map key / nil map value was dropped during canonicalization
— #{nil 1} canonicalized like #{1} and collided as a map key. So
(count {#{nil 1} :a, #{1} :b}) was 1 and (get {#{nil 1} :a} #{1}) was :a.
Box a nested nil before it goes into the table. The marker has to be
value-hashable, not the identity-hashed mutable-table sentinel the
transients use: the canonical struct becomes a long-lived phm key, and
its hash must survive the marshal/snapshot/fork that init-cached relies
on — an unmarshalled table gets a fresh address, so its hash isn't
preserved and the map can't find its own key. An interned keyword hashes
by content. Collision risk is only a real value equal to that exact
keyword, the same negligible class as canon-key's existing set/map struct
aliasing.
The transient sentinel stays a mutable table (it's built and consumed
within one op, never crossing a marshal boundary, so identity hashing is
stable there).
Co-authored-by: Yogthos <yogthos@gmail.com>
Two paths dropped a nil set member. phs-seq read members via
phm-to-struct, whose Janet struct can't hold a nil key, so the nil was
lost on seq/print and on into-an-existing-set even though the backing phm
counted it (count/contains? then disagreed). Re-attach the nil from the
phm's has-nil slot, keeping struct-key order for the rest so set printing
stays stable.
The transient set keyed its native table by canon-key and stored the
member as the value; canon-key returns nil for nil and a nil value also
drops the entry, so conj!/disj!/contains?/persistent! lost a nil member
outright. Key by tbl-key (the same nil sentinel the transient map uses)
and box a nil value through tbl-nil-key, unboxing on persistent!.
A nil-containing set used as a map key still collides with the nil-free
one (canon-key drops nil during key canonicalization) — separate latent
bug, filed as jolt-zcm9.
Co-authored-by: Yogthos <yogthos@gmail.com>
The map build already used a transient map, but each bucket was rebuilt with
a persistent (conj (get ret k []) x) per element — an O(log n) trie path
rebuild + alloc each. A coarse grouping (few large buckets) was bound on that
conj, not the map build. Buckets are now native arrays (transient vectors,
O(1) push) frozen once; distinct keys are tracked in a side vector so the
buckets freeze in place with no second map rebuild. A bucket's first element
stays a cheap persistent [x] and only promotes to a transient on the second,
so an all-singletons grouping pays no transient alloc.
coarse (10/100 buckets, 50k): ~313ms -> ~125ms (~2.5x)
2 buckets (50k): ~322ms -> ~129ms (~2.5x)
all-unique (50k): ~949ms -> ~892ms (no regression)
Surfaced a latent bug: canon-key returns nil for a nil key and Janet tables
drop a nil key, so the canon-keyed transient map silently lost a nil-key
entry — group-by/frequencies/assoc!/into{} dropped the whole nil bucket
((group-by identity [nil nil 1]) gave {1 [1]}, not {nil [nil nil], 1 [1]}).
Route nil through a sentinel (tbl-key) at the transient-map keying sites;
persistent!/count/dissoc! work unchanged since the real [nil v] pair is kept
as the stored value, and phm already has its own has-nil slot. The transient
set has the analogous bug (needs phs nil support) — filed separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
into {}, frequencies, group-by, set, into #{} and persistent! all built
their result by folding an immutable assoc/conj per element — each call
rebuilt the O(log32 n) trie path and allocated a fresh wrapper. Add a
one-pass bottom-up HAMT builder (phm-from-pairs) and route the builders
through it, the map/set analog of the pvec bulk build in #153.
phm-from-pairs partitions entries by hash and constructs the bin/array/
collision nodes directly, with the same bin<=16 / array-node>=17 promotion
the incremental path uses — so the trie is byte-identical to one built by
phm-assoc (validated across the size and branching boundaries, including
hash collisions, duplicate keys and the nil key). persistent! map/set and
the set constructor bulk-build; into {} keeps the small-scalar-map-stays-a
-struct rule via bulk-map-from-pairs; frequencies/group-by switch to the
canonical transient form and ride the fast persistent!.
50k A/B: into {} 704->270ms, frequencies 582->160, set 615->241,
into #{} 702->240, group-by 1358->919 (bound on persistent vector conj).
Gate: conformance x3, full suite (4718 >= baseline), new maps/sets bulk
boundary specs.
Co-authored-by: Yogthos <yogthos@gmail.com>
pv-from-indexed (behind vec/mapv/filterv/into-a-vector) built a pvec by calling
pv-conj once per element — each call allocated a fresh pvec wrapper and copied
the up-to-32 tail tuple, so building an n-vector was O(n) allocations + tail
copies. Replace it with a single bottom-up trie construction: chunk the elements
into 32-wide value leaves, group nodes 32-wide up to the root, split off the
tail. The structure is identical to the incremental one — tail-offset(n) =
((n-1)>>5)<<5 is exactly the trie/tail boundary, so nth/conj/assoc/seq read it
unchanged (validated against the old builder across the size boundaries).
into-a-vector likewise stops doing a persistent pv-conj per element: it
accumulates into a native array and bulk-builds once (the transient-style path).
Measured (50k): vec 211 -> 6 ms (~36x), into [] 197 -> 15 ms (~13x). mapv is
unchanged here — it's bottlenecked on lazy map realization, not the build.
The map/set builders (into {}, frequencies, group-by, set — all HAMT-backed)
need the same bulk treatment and are a separate follow-up. Gate: conformance x3,
full suite, new bulk-boundary rows in vectors-spec.
Co-authored-by: Yogthos <yogthos@gmail.com>
Add a PNG writer so the demos produce actual images. Two pieces:
- src/jolt/png.janet — the encoder (8-bit RGB, filter None, stored/uncompressed
DEFLATE so no compressor is needed; correct CRC32 + Adler32). It lives in Janet
because per-byte work in the overlay is far too slow (a byte-array aset loop is
~30s for 360k bytes, and CRC32 over even a tiny image would be worse). Janet's
bit ops are 32-bit signed, so the 32-bit-unsigned arithmetic is done with plain
number ops (doubles hold 2^32) plus byte-level bxor. Exposed to the overlay as
janet.png/* by importing it into eval_base's module-load-env.
- src/jolt/jolt/png.clj — the jolt.png overlay wrapper: image / put! / write. The
overlay only produces pixels; the host encodes them in one pass.
mandelbrot gets a `render` subcommand (jolt -m mandelbrot render out.png [size])
that colours count-point's escape counts; the numeric-arg bench path is untouched.
Verified end to end: macOS `sips` accepts the output (so CRC/zlib are valid).
png-test covers the encoder structure (signature/IHDR/IEND) and the overlay
round-trip.
Co-authored-by: Yogthos <yogthos@gmail.com>
emit-loop compiled every loop/recur to a self-recursive local closure called once
per iteration — relying on Janet TCO for stack safety but paying a fn frame + arg
bind each iteration. The jolt-5vsp spike localized the whole ~1.43x
jolt-over-hand-Janet gap on compute loops to exactly this.
Lower instead to a Janet `while` + state vars: the loop bindings become vars
carried across iterations, a recur writes them and raises a continue flag, and a
non-recur tail value falls out through a result var. recur-name routing in
emit-recur picks the while-set lowering for loops and leaves the fn-arity self-call
path untouched.
The one subtlety is closure capture: Janet closures capture vars BY REFERENCE, so
a closure built in the body over a shared mutable loop var would see the final
value ([3 3 3]) instead of its iteration's ([0 1 2]). Each iteration rebinds the
loop names into a fresh immutable `let` before running the body, which restores
per-iteration capture. recur reads those immutable bindings and writes the state
vars, so cross-referencing args (swap, fib) need no temps.
mandelbrot 218 -> 164 ms (~11.2x JVM, from 15x). fib is unaffected — it's
fn-arity recursion, not a loop. Regression spec in control-flow-spec covers
closure capture, no-clobber recur, nested loops, sequential init, recur through
let, and that fn-arity recur still works. Gate green (conformance x3, full suite).
Note: validating this after a rebuild needs JOLT_NO_DEPS_CACHE=1 — the deps-image
cache keys on the version string, not build identity, so it served stale codegen
(filed separately).
Co-authored-by: Yogthos <yogthos@gmail.com>
Fuse an app's native-compiled numeric-leaf fns plus its source into one
static executable: no sidecar .so, no toolchain on the target. The AOT path
(#148) already produced a prebuilt module + manifest; this links them into
the jpm-built exe so the app ships as a single file.
`jolt cgen-build -m NS -o OUT` stages a build dir (src/jolt-core symlinks
into the jolt tree, a generated cg.c of the hot fns, an uberscript bundle of
the app, and an entry that bakes the runtime, installs the native fns as var
roots, and runs -main), then runs `jpm build` there — declare-native builds
cg.a and declare-executable static-links it (jpm's create-executable marshals
the module cfns and calls its static entry at startup).
Build needs cc + jpm; the result needs neither. Mechanics that bit, codified
in cgen_build.janet: stdlib_embed slurps .clj cwd-relative so the build runs
in a repo-mirroring dir; jpm hardcodes ./project.janet and sets syspath=modpath;
the executable's dofile imports cg and static-links cg.a, neither ordered nor
release-built by default, so deps are wired explicitly; cleanup must lstat (the
tree symlinks must not be followed); the inner build runs --workers=1 so it
doesn't starve siblings in the parallel gate.
test/integration/cgen-build-test.janet builds the mandelbrot fixture, runs it
from a clean dir with no src/ and no cg.so, and checks the total at native
speed. Closes jolt-a7ds.
Co-authored-by: Yogthos <yogthos@gmail.com>
Splits native codegen into a build phase (needs cc) and a deploy phase (none):
- gen-c-module/compile-module compile MANY numeric-leaf fns into ONE native
module (the AOT shape), generalizing the one-fn-per-.so JIT path.
- Backend :cgen-collect? records each numeric-leaf defn's IR while the app loads
as bytecode; cgen/aot-build compiles them into one module and write-manifest
persists {sopath, [{ns name sym}]}.
- Backend :cgen-prebuilt + cgen/load-aot: the deploy run loads the prebuilt .so
(via the native builtin, no cc) and installs each cfunction as the var root
with the same timing as the JIT path, so callers direct-link to native code.
- toolchain-available? no longer crashes when cc is off PATH (os/execute raises
on a missing exe) — a toolchain-less target now gets false.
Proven end-to-end in two processes (spike/native/aot-demo.janet): build with cc,
then deploy with cc removed from PATH -> count-point still native, mandelbrot
3288753 at 12.4ms (full 18x). Test: test/integration/cgen-aot-test.janet. Default
path unchanged; the modes are opt-in. Gate green (118 files).
Remaining for a literal single binary: fuse the .so + manifest into the jpm exe.
Co-authored-by: Yogthos <yogthos@gmail.com>
compile-fn now keys the .so on a hash of the generated C + the Janet ABI + the
platform, in a persistent cache dir (default jolt-cgen under TMPDIR, override
with JOLT_CGEN_CACHE_DIR; JOLT_CGEN_NO_CACHE=1 forces a rebuild). cc runs only on
the first build of a given fn; later runs with the same source reuse the cached
.so, so the per-startup compile cost is paid once.
mandelbrot 100 whole-process wall: cold ~0.71s -> warm ~0.21s (the ~0.5s cc
cost). These cache knobs don't shape output, so they stay out of
ctx-shaping-env-vars (same as the image-cache knobs). Test asserts the .so is
content-addressed and a second compile hits the cache without the source .c.
Co-authored-by: Yogthos <yogthos@gmail.com>
Wires src/jolt/cgen.janet into the backend's :def emit. With JOLT_CGEN=1 (off by
default, needs direct-linking), a plain defn of a numeric-leaf fn is compiled to
C at def time and the cfunction installed as the var root, so direct-linked
callers embed native code. The fn is not inline-stashed when cgen fires —
callers must call the C fn, not inline the bytecode body. ^:redef/^:dynamic stay
bytecode.
The leaf-first rule falls out: run calls count-point (a user var), so run isn't a
numeric leaf and stays bytecode, calling the native count-point over the cheap
forward crossing. mandelbrot 200: 224ms -> 12.4ms (~18x), result unchanged.
Adds JOLT_CGEN to ctx-shaping-env-vars (rides the disk-cache key) and :cgen? to
resolve-run-mode. Default path (cgen off) is a no-op: cgen-root returns nil and
the normal bytecode emit runs. Gate green (117 files). Test:
test/integration/cgen-pipeline-test.janet.
Co-authored-by: Yogthos <yogthos@gmail.com>
First slice of the native-codegen tier. A new standalone module, src/jolt/
cgen.janet, that translates a numeric-leaf fn (numeric in/out, body uses only
native-op arithmetic + loop/recur/if/let/do) to a Janet native C module: params
unboxed to C doubles at entry, loop/recur lowered to a while loop, reboxed at
return. compile-fn runs cc and loads the .so via the native builtin, returning a
cfunction; it returns nil for non-candidates or when the toolchain is absent.
count-point compiles and matches the bytecode fn across the mandelbrot grid
(test/integration/cgen-test.janet, which skips the behavioral leg where cc/janet.h
are missing). Nothing wires this into the default compile path yet — detecting
hot fns and installing the C version onto the var cell is the next step.
See docs/foundational-runtime-lever1-native-codegen.md for the ceiling
(native-C ~18-22x faster than bytecode, edges out JVM) and the leaf-first rule.
Co-authored-by: Yogthos <yogthos@gmail.com>
run-tests.janet runs the same file set as `jpm test` across a pool of worker
processes (one `janet FILE` each, ev-based). The full gate goes from ~790s
serial to ~98s here (8x), and more on CI where the heavy files don't thrash on
swap. CI and the docs point at it; `jpm test` still works serially.
Three things dominated the wall:
- Nine integration tests cold-built a compile ctx (~8s each); switch them to
api/init-cached so they share the prebuilt image. The cache key already
fingerprints the ctx-shaping env vars, so the direct-link ones share one DL
image and the rest share the plain one.
- core-bench's main ran on every gate (~35s of benchmark loops that assert
nothing); gate it behind JOLT_BENCH=1.
- cli-test spawned `janet src/jolt/main.janet` ~20 times at ~8s cold each
(340s under parallel load, and it was the whole wall); prefer build/jolt
(~20ms baked ctx) when present, fall back to from-source for an unbuilt tree.
type-check-test stays on cold init: a snapshot-loaded ctx loses the success
checker's op/msg detail (jolt-vley). jolt-pria tracks caching from-source
startup generally, which would let cli-test drop the build/jolt preference.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Bind *command-line-args* after the deps-image cache swap (jolt-4mui)
Under whole-program (deps-image cache active), `jolt -m NS ARG` dropped ARG:
run-main set *command-line-args* on the current ctx, but a cache HIT then
replaced ctx with the saved image (via `set ctx cached`), whose *command-line-
args* was whatever got baked when the image was saved. The stale binding won at
`(apply NS/-main *command-line-args*)`, so -main ran with the wrong (usually
default) args — silently, for any optimized -m program.
Move set-command-line-args to AFTER the cache swap so it binds on the final ctx.
Repro/regression in deps-cache-args-test.janet: first run builds the image
(arg "first"), second run (cache hit) must echo "second", not the baked "first".
* docs: RFC 0003 — phm is a HAMT, sorted colls a red-black tree
The transients RFC described phm as "bucket-based copy-on-write" and mused about
"if it ever becomes a HAMT" — it is one now (jolt-684u), and sorted maps/sets are
a red-black tree (jolt-0hbr). Update the deviation/future-work notes accordingly.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Sorted collections were a sorted VECTOR — insert-at = (into (conj (subvec es
0 i) x) (subvec es i)) is O(n) per assoc with a large constant, so building was
O(n^2): 2000 entries took 55.6s.
Replace the rep with a red-black tree (assoc/dissoc/get/contains O(log n)),
ported from the ClojureScript PersistentTreeMap (cljs.core: tree-map-add /
balance-left / balance-right / tree-map-append / balance-*-del). This tier (25)
loads before 30-macros so deftype isn't available; a node is a plain vector
[color k v left right] and cljs's BlackNode/RedNode methods become functions —
the algorithm is unchanged. A sorted-set stores elements as keys with a nil
value; its ops project the key.
The seed read the old :entries vector directly for equality/printing; route
those through a new :entries op that materializes ascending from the tree
(core_types/sorted-entries-arr + main.janet's printer).
2000 sorted-map assocs: 55.6s -> 0.98s (57x); now O(log n) (per-op cost flat
from n=2000 to 10000). Correctness in test/integration/sorted-rbtree-test.janet
(shuffled insert ordering, delete rebalancing, custom comparator, comparator
lookup, subseq, count); sorted specs + full gate green. (key/val on sorted
entries stays a pre-existing gap — entries are pvecs not host tuples; jolt-jk23.)
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add benchmark suite for alloc/dispatch/collection workloads (jolt-1r86)
The ray tracer is float-compute-bound (devirt, alloc removal, type-proving all
measured flat on it), so it can't validate the optimization passes. Add a small
cross-language suite (AWFY + CLBG style, portable Clojure) isolating the axes it
misses:
binary-trees allocation / GC pressure (escaping short-lived records)
dispatch megamorphic protocol dispatch (~1M dispatches/s; WP can't devirt)
collections persistent map/vector churn
bench/run.sh runs them; bench/README.md maps each to the pass it exercises.
collections immediately surfaced jolt-684u: the persistent hash map is O(n) per
assoc (flat copy-on-write bucket array, not a HAMT) — n=4000 assocs take 50s.
Invisible to the ray tracer (no maps).
* Persistent hash map: HAMT instead of O(n) copy-on-write (jolt-684u)
The map was a flat bucket array whose assoc copied the whole array every insert
(O(n)/assoc, O(n^2) to build). Compounding it, small maps are Janet structs that
only promoted to phm for collection keys — never for size — so a scalar-key map
stayed an O(n)-copy struct forever. Building a 4000-entry map took ~50s.
Two fixes, following ClojureScript's design:
- phm.janet is now a HAMT (hash array mapped trie): BitmapIndexedNode /
ArrayNode / HashCollisionNode, 32-way, 5 hash bits per level, structural
sharing — assoc/dissoc/get are O(log32 n). Translated from cljs.core, adapted
to Janet's 32-bit bit-ops (the hash is carried unsigned, the level index is
extracted with arithmetic, and bits are tested with band against 1<<i since
brushift rejects negative bitmaps). The public phm-* API and the value shape
(:jolt/type :jolt/phm, :cnt) are unchanged; transients are a separate rep and
untouched.
- core_coll promotes a struct map to a phm past 8 entries (not only for
collection keys), mirroring cljs PersistentArrayMap -> PersistentHashMap, so
incremental building isn't O(n^2).
20000 raw assocs: 7.1s -> 0.105s. The collections benchmark: 16.7s -> 0.2s.
Correctness covered by test/unit/phm-hamt-test.janet (oracle vs a Janet table,
nil keys, dissoc, a real hash-collision pair, and a sub-linear-assoc guard);
full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
A ^Record param hint was applied only at the final re-emit (reinfer-def), not
during the inter-procedural fixpoint. So a hinted param with no callers stayed
:any while inference ran, and a field read off it (e.g. (:origin ^Ray r)) never
told a non-inlined callee that its arg is a Vec3 — the callee's params stayed
unproven and its field reads kept the dynamic guard.
Seed declared hints as a param-type floor in the fixpoint: phint-seed (passes/
types) resolves an arity's :phints to positional record types via the
record-shapes registry, and infer-unit! initializes each fn's fresh param slots
from them instead of nil. A fixed declared type can't poison the least-fixpoint
the way an early-iteration :any would, and a hinted param now propagates its
(and its field reads') types to its callees during inference.
Scope: this closes the hinted-propagation gap. It does NOT help the ray tracer,
which uses zero ^-hinted params (only hinted fields) — its remaining type gap is
unhinted record-param inference on recursive/non-inlined hot fns, and per the
jolt-15jq A/B it's allocation-bound regardless (jolt-8flj). Tracked on the bead.
Co-authored-by: Yogthos <yogthos@gmail.com>
Dependency resolution now lives in the `jolt` CLI itself instead of a separate
jolt-deps executable. `jolt` resolves a deps.edn into JOLT_PATH/JOLT_APP_PATHS
in-process and dispatches the deps subcommands:
jolt -M:alias [args] run the alias :main-opts
jolt -A:alias CMD run CMD with the alias paths
jolt run FILE resolve, then run FILE
jolt path | tasks | task NAME
A deps.edn in the working dir is auto-resolved for the runnable commands
(repl/-m/-e/nrepl-server/FILE), so e.g. `jolt -M:nrepl` (or plain
`jolt nrepl-server`) starts an nREPL with the project and its deps loaded.
The runtime core stays deps-agnostic — it only reads JOLT_PATH. The resolver
(deps.janet) is reached only from the CLI entry and loads jpm lazily, so a run
with no deps.edn never touches it and an app baked from its own jolt/api entry
never links it. resolve-deps-argv only resolves on an explicit deps command or
when a deps.edn is present; help/version never do.
jolt-deps stays as a thin deprecation shim that forwards to `jolt`, so existing
scripts keep working. Docs (README, CLAUDE.md, building-and-deps, tools-deps)
and the help text updated.
Co-authored-by: Yogthos <yogthos@gmail.com>
scalar-replace already folds non-escaping const-key map literals
((:k {:k a ..}) -> a, and drops a let-bound map that doesn't escape).
Extend the same fold to record constructors: a (->Rec a b c) is a
positional struct whose declared field order lives in the record-shapes
registry, so a field read on a non-escaping ctor folds to the matching
positional arg and the allocation disappears.
Direct form (:field (->Rec ..)) and the let-bound form both handled,
threaded through run-passes via a per-unit shape registry (new
jolt.host/record-shapes accessor). Soundness: ctor args must be pure
(duplicated/discarded like map vals), arg count must equal the field
count, and only declared-field reads fold — a record answers the virtual
:jolt/deftype key with its type tag and any other key with nil, neither
of which is a positional arg, so those keep the allocation. pure? now
treats a record ctor of pure args as pure, so nested records (a Ray
holding a Vec3) fold bottom-up.
Allocation-bound microbench (non-escaping record built + field-read in a
hot loop): 69.6s -> 2.4s, landing on the no-record arithmetic baseline.
The ray tracer is unchanged — its vec3 results escape (returned/stored
each op), so they genuinely allocate; that's a separate problem.
Co-authored-by: Yogthos <yogthos@gmail.com>
cap truncates a deep type's field VALUES to :any so the inter-procedural
fixpoint stays finite, but it rebuilt the struct via mk-struct and dropped the
record :type tag along the way. The tag is identity — independent of field
depth — so a record stored in a deep container (a Sphere in a world vector, a
material on a hit) degraded to a plain struct, and devirtualization (jolt-41m)
and record? folding silently stopped firing on it.
Preserve :type alongside :shape when capping. Verified: a protocol call on a
record read out of a vector now devirtualizes (the call node gets :devirt-type,
which needs the receiver's record type). Sound — the tag stays accurate; only
field values below the depth cap are truncated.
No measurable wall-clock change on its own (jolt's protocol dispatch is already
cheap), but it restores the record fast path / devirt / record?-folding on
records-in-containers, and unblocks downstream work that keys off record types.
Co-authored-by: Yogthos <yogthos@gmail.com>
A bundle is closed-world — everything it needs is inlined and nothing is
required afterward — so a user defn unreachable from the entry's reference
graph can be dropped. The bundler now computes reachability from main-ns/-main
plus every non-prunable form and drops dead defn/defn- by exact source span
(formatting and reader macros in the surviving code are untouched).
Conservative and sound: only plain defn/defn- are prunable; a defn is kept if
its bare or ns-qualified name appears in any kept form, the closure runs to a
fixpoint, and any use of resolve/ns-resolve/requiring-resolve/find-var/intern/
eval/load-string disables pruning entirely. A parse failure on any file also
falls back to verbatim bundling, so the command stays as robust as a plain
concatenation. defmethod/defrecord/extend bodies are non-prunable and scanned,
so a fn reached only via dynamic dispatch stays live.
New reader/parse-all-spans returns [form start end] byte offsets so the drop
is a verbatim slice, not a re-print.
30-fn library used by a 3-fn entry: bundle 1114 -> 437 bytes (61% smaller, 27
dead fns dropped), output byte-identical.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the collection-type inference proves an argument's type, number?/
string?/keyword?/record?/nil?/some? fold to a compile-time boolean. A
const-fold now runs after inference so a folded predicate propagates and
collapses any if it gates to the taken branch.
Sound by construction: only a provable answer folds, and only when the
argument is side-effect-free (a const or local) so dropping its evaluation
is a no-op. Unknown types (:any/:truthy) and impure args keep the call.
vector?/set?/map? are left out — the :vec tag conflates a real vector with
a range/seq, so vector? could be wrong.
50M-iter loop, same shape isolated with a carry-only control: number? call+
branch 5080ms, predicate folded 1365ms — matching the 1417ms control floor,
so the 3.7x is entirely the eliminated call+branch.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Reader: #() params survive syntax-quote (auto-gensym names)
#(...) named its synthesized params with bare gensyms, so a #() written inside a
syntax-quote had its params qualified to the current ns by sq-symbol — and a
qualified symbol isn't a valid fn param. hiccup's compiler emits
`(let [sb# ..] (iterate! #(.append sb# %) ..)), which broke with "Unable to
resolve symbol: ns/_NNNN".
Name the params with a trailing # (auto-gensym suffix, like Clojure's p1__N#) so
syntax-quote maps them consistently and leaves them unqualified. Harmless outside
a backtick (just a regular symbol name).
* interop: String/valueOf static + String is a CharSequence
Two interop gaps surfaced bringing up hiccup and malli:
- String/valueOf(Object): hiccup's compiler stringifies attribute values with
(String/valueOf (or arg "")). Added the static — "null" for nil, else core-str.
- (instance? CharSequence s) returned false for a string; String implements
CharSequence, and malli's :re validator gates on it before matching, so :re
schemas always failed. instance-check now answers true for strings.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Under JOLT_OPTIMIZE a -m program run inferred + specialized EVERY loaded
namespace, including every transitive dependency. On a dep-heavy app that's
prohibitive: malli-app cold-started in ~2m10s (hundreds of dep namespaces, each
run through the per-form inline + inference passes).
The closed world a whole-program pass reasons over is the APP, not its
libraries. jolt-deps now passes the project's own source roots (its deps.edn
:paths) to the runtime as JOLT_APP_PATHS. A namespace loaded from an app root
gets full optimization (and joins the one whole-program fixpoint); a dependency
namespace compiles at default cost — :inline? off for its load, so the per-form
optimize passes don't run over library code — staying direct-linked but
generically typed (the open-world default). With no app roots declared (a bare
program run, or jolt without jolt-deps) everything counts as app, so behavior is
unchanged.
malli-app JOLT_OPTIMIZE cold start: 2m10s -> 4.5s. Compute-heavy programs whose
hot code is their own namespaces (the typed ray tracer) are unaffected — their
code is app code and still fully optimized (9s/frame render). Applied at runtime
in main for the same baked-at-build-time reason as JOLT_PATH; added to the
ctx-image cache key. Help text corrected: optimization is opt-in, not default.
Co-authored-by: Yogthos <yogthos@gmail.com>
