The transient assoc! accepted an odd key/val count and silently assigned
nil to the dangling key — non-Clojure, and inconsistent with the seed's
own plain assoc (which throws) and Clojure's assoc!. Make it throw.
Updates the 4 'assoc! odd args' transient spec rows to 3 :throws + 1
even-args positive, and regenerates corpus.edn. The Chez host already
threw on these, so this only realigns the corpus contract.
host/chez/dynamic-vars.ss binds the two seed-native dynamic vars that aren't
emitted into the prelude: *clojure-version* (the {:major 1 :minor 11 ...} map) and
*unchecked-math* (false). Removes their two run-corpus-prelude allowlist entries
(now 11, all passing).
*ns* is deferred to jolt-b4kl: it needs a namespace value that is not a map yet
answers (get ns :name) for the overlay ns-name, plus str/find-ns support.
Parity 1530 -> 1532/2497, 0 new divergences. emit-test 305/305.
host/chez/multimethods.ss implements the multimethod runtime: defmulti/defmethod
expand to defmulti-setup/defmethod-setup calls (+ get-method/methods/
remove-method/prefer-method/prefers). A jolt-multifn record carries its dispatch
fn and a jolt=-keyed method table; jolt-invoke dispatches it (exact match, then
isa?/hierarchy with prefer-method, then :default), reusing the overlay's
isa?/derive/make-hierarchy. The multifn's ns comes from a runtime chez-current-ns
(default user; the prelude load sets clojure.core for print-method/print-dup).
Two emit-side changes were needed:
- late-bind (:late-bind-unresolved? ctx flag, default OFF): defmulti expands to a
bare-symbol setup call, so the analyzer doesn't intern the name and a forward
reference '(area ...)' after '(defmulti area ...)' in one form was 'Unable to
resolve symbol'. The strict compiler punts these to the interpreter; the Chez
back end has none, so the flag lowers an unresolved symbol to a var-ref against
the compile ns (open-world -e semantics). Set only by the Chez make-ctx /
jolt-chez; the main compiler keeps strict resolution (host_iface late-bind?
defaults nil).
- a :var call head now routes through jolt-invoke, since a late-bound var can hold
a multifn (or keyword/coll IFn), not just a procedure. Transparent for
procedures; the hot self-recursive call is a :local known-proc, stays direct.
Class-based dispatch ((class x)/String) deferred (needs deftype/class subsystem).
Parity 1506 -> 1530/2497, 0 new divergences. emit-test 302/302. Full janet gate
green (the analyzer flag is off there; suite flakiness under parallel load only).
jolt-y1zq tail:
- 0-arg (conj) -> [] and 0-arg (conj!) -> a fresh transient vector
- nth sees through a transient (like get/count/contains?)
- irregex \p{...}/\P{...} property classes translate to the seed's ASCII char
classes (regex.ss): \p{L} -> [a-zA-Z] + non-ASCII codepoints (the seed counts
UTF-8 high bytes as letters), \p{N} -> [0-9], \p{Ps} -> [([{], etc. The
translator tracks [...] nesting so a \p{} inside a class emits its content, not
a nested class.
Two pre-existing bugs found and filed (tracked, not replicated):
- jolt-x0os: the Chez emitter mangles non-ASCII string literals into invalid Chez
hex escapes (so p{L} utf-8 crashes on the input string, not the regex).
- jolt-ea9k: the seed's transient assoc! accepts odd args and assigns nil to the
trailing key (non-Clojure; plain assoc throws). The Chez host throws
(Clojure-correct); the 4 spec rows encoding the leniency are flagged in
transients-spec.janet pending the seed fix.
Parity 1493 -> 1506/2497, 0 new divergences. emit-test 291/291.
The 1-arg map/filter/remove/take/drop/take-while/drop-while/mapcat now return a
transducer (fn [rf] rf'), and into gets a 3-arg (into to xform from). This was
the 'cdr () is not a pair' / 'incorrect number of arguments' crash bucket: the
emitter lowers (map f) and 3-arg into at an arity the native-op gate rejects, so
they fall to the value-position path and hit the bare jolt-map/jolt-into
procedure at the wrong arity. The fix is RT-side — case-lambda those procedures
plus jolt-into.
td-* factories ported from the seed (core_coll.janet); a reduced step stops the
fold via reduce-seq's existing short-circuit (inc 3n). transduce/comp/completing
are overlay and compose over these unchanged.
Parity 1467 -> 1493/2497, 0 new divergences. emit-test 278/278.
The dominant prelude-parity crash bucket was 'apply non-procedure jolt-nil':
core fns calling seed-native seq fns (core_coll.janet) that have no Chez RT
shim, so var-deref returns jolt-nil. A static scan of the assembled prelude
turned up 52 referenced-but-undefined clojure.core names.
host/chez/natives-seq.ss shims the safe seq fns over the existing seq layer:
mapcat, take-while, drop-while, partition (collection arities only — the 1-arg
transducer forms are jolt-kxsr), and sort (compare default; a comparator may
return a 3-way number or a boolean less-than). reduced/reduced? is a jolt-reduced
record in seq.ss that reduce short-circuits on and deref unwraps, so unreduced
works. identical? = jolt= (the seed's definition).
Deferred list?: a Chez lazy seq and a list are both cseq, so it can't be told
apart without a distinct list type — a real divergence risk.
Parity 1407 -> 1467/2497, 0 new divergences. emit-test 263/263.
##Inf/##-Inf/##NaN were emitted to bare inf/-inf/nan, which are unbound symbols in
Chez. emit-const now lowers them to +inf.0/-inf.0/+nan.0. The -e/element printer
renders them inf/-inf/nan (Chez's number->string gives +inf.0), and str renders the
long Clojure forms Infinity/-Infinity/NaN. assoc! is now variadic ((assoc! t k v
& kvs)) like Clojure.
Prelude parity 1382 -> 1407/2497, 0 new divergences. str of inf INSIDE a collection
still wants the long form (needs the Phase-2 recursive str renderer), so
[inf inside coll] is allowlisted. Transducer arities and the cdr-on-()/\p{} regex
gaps are split out to jolt-kxsr/jolt-y1zq.
transient/persistent!/conj!/assoc!/dissoc!/disj!/pop! as copy-on-write over the
persistent collections (host/chez/transients.ss) — each op rebuilds the persistent
coll (no in-place perf) but the semantics match, so into/frequencies/group-by work.
Adds persistent disj over pset-disj. get/count/contains? are redefined to see
through a transient (frequencies and group-by both do (get tm k) on a transient
map); vector? on a transient vector is false, which group-by relies on.
Prelude parity 1326 -> 1382/2497, 0 new divergences. emit-test exercises the direct
transient ops via run-prelude and the overlay users (frequencies/group-by/into)
end-to-end through the bin/jolt-chez -e binary.
str/subs/vec/keyword/symbol/compare/int/double/gensym — host-coupled seed natives
the overlay assumes, now def-var!'d into clojure.core via host/chez/converters.ss
(loaded last, str reuses jolt-pr-str). Semantics match the seed: str-render-one
for str (nil->"", bare chars, regex source), the 3-way core-compare port, int
truncates. The symbol no-ns sentinel is #f to match emit's quoted-symbol lowering
(jolt-symbol #f "x"), so (= 'x (symbol "x")) holds — jolt= compares ns with strict
equal?, and jolt-nil vs #f would otherwise diverge.
Prelude parity 1220 -> 1326/2497, 0 new divergences. Floor raised to 1326; three
newly-reached *ns*/var-rendering cases added to the allowlist (Phase 2). run-prelude
and the per-case program file are now PID-unique so concurrent chez runs don't read
each other's half-written files.
Emit every non-macro clojure.core form through the live analyzer -> Chez emit
pipeline as a def-var! prelude (prelude mode, tier dependency order), load it
before a user expression, and you get an -e-capable jolt-chez: analysis on Janet,
execution on Chez. driver/emit-core-prelude assembles it (each form behind a
silent load guard so the Phase-2 multimethod forms don't break the rest);
bin/jolt-chez is the -e CLI, caching the prelude on disk keyed by source hash.
run-corpus-prelude.janet is the full parity gate this opens, the prelude-backed
sibling of run-corpus-chez. First baseline: 1220/2497 evaluated cases pass, 0 new
divergences (10 allowlisted: dynamic vars, class names, eval-order — deferred
Phase 2). The rest is the punch-list: ~360 emit-fail (real host interop, out of
the analyzer subset) and ~900 runtime crashes, mostly core fns calling
host-coupled seed natives with no Chez shim yet (str/format/vec, transients).
Follow-ups jolt-t6cr/kl2l/q3w8/9ls5.
Two shims landed to get the prelude to load and run. atoms.ss: atom/deref/swap!/
reset! (+ the compare/vals kernel) — needed at load time for
global-hierarchy = (atom (make-hierarchy)). predicates.ss: the type predicates +
name/namespace/boolean the overlay assumes are seed natives, matching the seed's
strict semantics. post-prelude.ss re-asserts char?/atom? after the prelude: the
overlay implements those by reading :jolt/type, which is false for Chez-native
chars/atoms, so its def-var! would clobber the correct native versions.
Per-case Scheme files are PID-unique so a foreground -e never reads a half-written
file while the gate runs.
Closes the last clojure.core prelude emit gap (parse-uuid): the whole
non-macro core now lowers to Scheme (prelude reach 355/355).
A #"..." literal analyzes to a :regex IR node. The Chez back end emits
a jolt-regex value over irregex (Alex Shinn, BSD), vendored as the
vendor/irregex submodule -- a portable Scheme regex with PCRE/Java-style
string patterns and first-class Chez support. host/chez/regex.ss wraps
jolt's re-* surface over it: irregex-match -> re-matches (anchored),
irregex-search -> re-find, groups as Clojure [whole g1 ...] vectors,
re-seq as a jolt seq. re-pattern/re-matches/re-find/re-seq/regex? are
def-var!'d into clojure.core so prelude / -e code resolves them.
They stay OUT of the subset native-ops on purpose: irregex's
Unicode/property-class semantics differ from the seed's byte-PEG
approximation, so keeping them prelude-only avoids dragging
engine-difference divergences into the subset-parity corpus. The Janet
back end punts :regex to the interpreter (the seed compiles #"..." to a
Janet PEG), so the main language is unchanged.
Only two adaptations for Chez's top level: a cond-expand shim (Chez's is
library-only) and a normalizing error wrapper (silences irregex's 1-arg
error warnings). rt.ss load is ~0.18s.
emit-test 131/131 (regex literal + re-* parity vs the CLI oracle);
prelude reach 355/355; Chez subset 672/672, 0 divergences; full gate
green.
(.method target arg*) now analyzes to a :host-call IR node instead of
punting at analyze. The Chez back end lowers it to a jolt-host-call
dispatch for the methods the RT shims (.write -> port display,
.isDirectory -> file-directory?, .listFiles -> directory-list); any
other method stays out of subset (clean emit-time reject, so it can't
read as a compiled-but-broken corpus divergence). The Janet back end
punts ALL :host-call to the interpreter, same shape as letfn: compiles
on Chez, interprets on Janet, zero change to the main language.
Closes the io tier's print-method defmethods and file-seq: prelude emit
reach 348 -> 354/355 (50-io 20/20). The one remaining gap is the regex
literal in parse-uuid (needs a regex engine on Chez; deferred).
emit-test 122/122; Chez subset 672/672, 0 divergences; full gate green.
Closes the last two non-host-interop prelude emit gaps.
letfn now analyzes to a :let node flagged :letrec — the binding fns are bound
into the env together before any spec is analyzed, so siblings and self resolve.
The Chez back end lowers it to letrec*; the Janet back end punts it at emit
(its sequential let* can't express the mutual recursion — same interpreter
fallback as before, just decided at emit-ir instead of analyze).
(def x) with no init (declare) analyzes to a :def with :no-init instead of
punting. Chez reserves the var cell via declare-var! (which doesn't clobber an
existing root — (do (def x 7) (def x) x) => 7); the Janet back end still punts
to the interpreter, which interns a genuinely-unbound var.
fallback-zero-test now checks emit-ir too, not just analyze-form, so the real
compile-vs-interpret decision is what it asserts (letfn/def-no-init analyze but
the Janet back end punts them). letfn stays in must-punt with an updated note.
Prelude emit reach 342 -> 348/355 (40-lazy now 13/13); Chez subset 664 -> 672,
0 divergences; emit-test 110 -> 117. Full gate green.
Emit a :quote node by reconstructing the raw reader form as RT constructor
calls: symbol -> jolt-symbol, list (array) -> jolt-list, vector (tuple) ->
jolt-vector, map -> jolt-hash-map, set -> jolt-hash-set, scalars via emit-const.
The runtime value of a quote is just that literal data (the interpreter returns
the reader form verbatim).
Quote exposed a latent seq.ss bug: empty map/filter results were jolt-nil, but
Clojure's (map f []) is an empty seq, so (= () (map f [])) must be true. Return
jolt-empty-list (which seqs back to nil, so it's still a valid lazy-tail
terminator) instead — matching jolt-take/drop/rest/list.
Prelude emit reach 334 -> 342/355. Subset probe 632 -> 664/664 compiled, 0
divergences (quote + the seq fix pull 32 corpus cases into the subset). emit-test
110/110 (added 16 quote cases). corpus.edn regenerated (the 3 malformed-catch
spec rows). Full gate green.
jolt's catch is (catch class binding body*); the binding (3rd element) must be
a symbol. Neither the analyzer nor the interpreter validated it, so a non-symbol
binding crashed with an internal Janet error (expected integer key for array...)
and, in the interpreter, a malformed clause whose body never threw was silently
swallowed (returned the try value). Clojure rejects a non-class/non-symbol catch
clause; match that with an up-front error in analyze-try and eval-try.
Surfaced building the Chez try/throw emit. Regression rows in exceptions-spec
(runs x3 modes) plus a unit test asserting the clean message in interpret and
compile. jolt-kg6p.
Emit :throw as jolt-throw (Scheme raise of the raw jolt value, matching the
Janet compiled back end's (error v) — no envelope, so catch binds it directly).
Emit :try as guard (catch; the class is dropped in the IR, so it's catch-all)
plus dynamic-wind for finally. ex-info is a native-op building the tagged jolt
map {:jolt/type :jolt/ex-info :message :data :cause}, so the ex-data/ex-message/
ex-cause tier fns read it over jolt-get for free.
Prelude emit reach 303 -> 334/355 (:throw and :try gaps close). Subset probe
619 -> 632/632 compiled, 0 divergences (throw/try/ex-info pull 13 corpus cases
into the subset). emit-test 94/94 (added 11 throw/try/ex-info cases + uncaught
exits non-zero). Full gate green.
Add a prelude emit mode to host/chez/emit.janet: when emitting clojure.core
itself (not user -e), a non-native clojure.core ref lowers to a runtime
var-deref instead of being rejected as out-of-subset, so core fns chain
through each other. Default (subset) mode is unchanged — the corpus probe
still rejects unimplemented core refs for a clean signal.
core-prelude-probe.janet walks the tiers through the live analyzer->emit
pipeline and catalogs reach + gaps (macros skipped; analyze-time only).
Baseline: 303/355 non-macro core forms emit. Remaining gaps are a tight
punch-list for the next increments: :throw (29), :quote (8), :try (2), Java
host interop (6), letfn (4), declare (2). Probe has a regression floor.
emit-test 83/83 (added prelude-mode lowering assertions); subset probe
619/619 unchanged; full gate green.
emit-fn lowered multi-arity fns to a Scheme case-lambda and variadic fns to
a rest-arg lambda; the Scheme rest list is coerced to a jolt seq (nil when
empty, via list->cseq), and the named-let wrapper runs that coercion only on
first entry so recur carries the seq directly. Single fixed arity keeps the
plain-lambda fast path (fib untouched).
Also fixes a latent leak in the module-global known-procs: a throw mid-emit
(uncompilable body) left the fn's name registered, so a later corpus case
binding the same name to a keyword emitted a direct call to a non-procedure.
The cleanup now runs on the error path too. Only surfaced once the new arity
support let +24 cases compile further before hitting an uncompilable fn.
Gate: emit-test 81/81, subset probe 619/619 compiled (was 595), 0 divergences,
2036/2655 out of subset; full run-tests green (125 files).
Brings up the seq layer on the Chez runtime. host/chez/seq.ss adds one
lazy-capable node (cseq) that models Clojure's list, cons, and lazy seq -
all print as (...), all sequential-= to each other and to vectors. seq
coerces any seqable (vector/map/set/string/list/seq/nil) to a cseq or nil;
the empty seq is a distinct value printing () (rest of a 1-elem coll is ()
not nil, seq of empty is nil). Leaf ops: first/rest/next/seq/cons/list,
reverse/last, map/filter/remove/reduce/into, range/take/drop/concat/apply,
keys/vals, plus nth/peek/pop extended over seqs. map/filter/reduce apply
their fn arg through jolt-invoke, so a procedure, keyword, or collection all
work as the fn.
Dynamic IFn dispatch: a keyword/vector/coll held in a local (let binding or
fn param) and called as a fn now routes through the jolt-invoke fallback
(procedure? -> apply; keyword/coll -> lookup). The emitter only routes a
:local callee that isn't a known procedure - a named fn's self-recursion
name stays a direct call, so the fib hot path is untouched. Closes the 3
ex-known IFn divergences.
emit.janet: seq/pred ops added to native-ops with arity gates; value-position
clojure.core refs resolve to the RT procedure (native-ops names one for each),
with +/-/*// routed to flonum-coercing wrappers so higher-order arithmetic
((reduce + [])) keeps the all-double model. values.ss: cross-type sequential
=/hash so a vector and a list of the same elements are jolt= and hash alike.
rt.ss: printer learns seqs; top-level nil prints as the empty string (jolt -e
str-style). Fixed latent bug: (conj nil ...) now builds a list, not a vector.
Gates: emit-test 69/69 (fib/mandelbrot/collections/seq/IFn parity vs the jolt
oracle, fib(30) ~24ms unchanged). Subset probe 433/436 -> 595/595 compiled,
0 divergences (was 3 known), 2060/2655 out of subset. Full run-tests green
(125 files, conformance + suites included).
Broaden the Scheme back end past the numeric/functional subset to vectors,
maps, and sets. host/chez/collections.ss adds a copy-on-write persistent
vector and a bitmap HAMT (the structure 0c measured self-hostable) backing
both maps and sets, keyed by jolt-hash and compared by jolt=. emit.janet
emits :vector/:map/:set literals to the rt constructors and lowers the leaf
ops (conj/get/nth/count/assoc/dissoc/contains?/empty?/peek/pop) via the
native-ops path, with a per-op arity gate.
Also: keyword/map literals in fn position lower to jolt-get ((:k m), ({:k v} k));
arity-1 comparisons emit the vacuous jolt truth (Scheme < rejects a non-number
even at arity 1); count returns a flonum and vector indices coerce from flonum,
both consequences of the all-double number model; values.ss = / hash and the
rt printer learn collections (maps/sets render in HAMT order, so the probe
compares unordered values via =, not printed form).
Subset parity 182 -> 433/436 compiled cases (2219/2655 out of subset), 0 new
divergences. The 3 known divergences are dynamic IFn dispatch (a keyword/vector
held in a local, called as a fn) — deferred to the IFn/protocol increment and
allowlisted in the probe. emit-test 31/31, full run-tests green (125 files).
Wire the real pipeline end to end: host/chez/driver.janet boots a compile-mode
jolt ctx, runs the EXISTING Janet-hosted analyzer on actual Clojure source to
real IR, feeds it to the Scheme emitter, and runs the result on Chez. Analysis
stays on Janet (the analyzer ports to Chez in Phase 2); execution is on Chez.
emit.janet now consumes live IR (pv/phm-normalized like the Janet backend) and
covers what the analyzer actually emits, not the hand-built inc-1 shapes:
- core ops arrive as :var clojure.core/+ etc., not :rt — lowered to native
Scheme via a native-ops table (mirrors backend.janet's), `=` to jolt=.
- var cells (host/chez/rt.ss): :def -> def-var!, :var -> var-deref. Late binding
so cross-var calls (run -> count-point) and the entry crossing resolve at use.
- named fns (defn / fn self-name) bind via letrec so self-recursion resolves.
- unsupported stdlib/host refs (no core on Chez yet) are rejected at EMIT time
(clean out-of-subset signal) instead of deref'ing to nil and failing at runtime.
Number model: jolt is all-doubles (no ratios; (/ 1 2) is 0.5), so literals emit
as flonums — matches the Janet host and keeps Chez out of exploding exact
rationals (mandelbrot). jolt-num->string prints integer-valued without ".0".
Two real bugs found via the corpus probe and fixed (regression rows added):
- loop bound in parallel (Scheme named-let) but Clojure loop is sequential — a
later init must see earlier bindings; wrap a let* around the loop.
- #(...) shorthand gensyms params with a trailing `#`, invalid in Scheme — munge
it to `_`.
Gate: test/chez/emit-test.janet runs the real analyzer -> Chez for (+ 1 2),
fib(30)=832040, mandelbrot run(40), and the two regressions, parity-checked
against the Janet oracle (6/6). First parity number via the new subset probe
(test/chez/run-corpus-chez.janet, JOLT_CHEZ_CORPUS=1): 182/182 compiled corpus
cases pass, 0 divergences; 2473/2655 out of subset pending core on Chez. Full
jpm/run-tests gate green (125 files). Chez tests skip cleanly without `chez`.
Perf note (unchanged plan): emitted fib(30) ~23ms vs hand-Scheme ~5ms — the
jolt-truthy? wrapper (~3x) plus flonum (not fixnum) arithmetic, both Phase-4
type-specialization levers.
New back end half: host/chez/emit.janet consumes the host-neutral jolt IR
(ir.clj shapes) and emits Scheme, reusing the existing front-end (Option-2
backend swap). Covers the pure-functional subset: const/local/var/rt/if/do/let/
fn/invoke/def/loop/recur. Tested by hand-built IR run on Chez: (+ 1 2)=3,
fib(30)=832040, loop/recur sum=15 (4/4).
Finding: correct emit wraps every if-test in jolt-truthy?, costing ~3x on fib
(15.8ms vs hand-Scheme 5ms). Eliding the wrapper for known-boolean tests
recovers the ceiling (Phase-4 type-driven opt).
Remaining Phase 1: wire the live analyzer, var-cell late binding, RT module,
broader op coverage for mandelbrot.
0c: persistent HAMT on Chez is ~41x faster than Janet's HAMT on the collections
map-churn (258.6 -> 6.3 ms), ~15x off mutable-native (inherent persistence cost).
Decision: self-host the persistent collections in Clojure; substrate is not the
bottleneck. See docs/chez-phase0-results.md.
0a hardening: NUL-separated keyword intern key (no ns/name collision), non-finite
-safe jolt-hash. 37/37.
0a (host/chez/values.ss): Jolt value model on Chez — nil sentinel distinct from
#f/'(), interned keywords, ns+meta symbols, exactness-aware = and consistent
hash. Chez numeric tower gives ratios/bignums free. 33/33 tests.
0b (test/chez/): extract test/spec/*.janet defspec tables into corpus.edn (2655
cases, valid as both EDN and Janet data), and a runner that drives ANY jolt
binary via the CLI boundary with per-case subprocess isolation. Pluggable target
(JOLT_BIN) so the same corpus gates every host. Baseline vs Janet build/jolt:
2641/2655, 14 known CLI divergences allowlisted; gate fails only on NEW ones.
host.* replaces janet.* as the portable interop namespace (each host implements
it over its own FFI). Add a :native dep form so projects declare needed shared
libs (libcurl/openssl/zlib) — not git-fetchable, but surfaced to the user and
probed at load so a missing .so yields a precise error, not a raw dlopen fail.
Account for jolt's layered interop surface on Chez — the janet.* bridge, the
FFI-backed java.* shim libs (http-client TLS/gzip, router, db), jpm-module Janet
deps (spork/http) — with ../examples as the end-to-end acceptance gate. New
epic child jolt-cf1q.7, gated behind Phase 2.
Phased plan for re-hosting jolt's substrate on Chez Scheme, organized around two
north stars: minimal host shim (push everything possible into self-hosted
jolt-core, drop the tree-walking interpreter) and the spec/conformance corpus as
the host-neutral correctness contract. Closes obsolete Janet-backend/cgen beads
superseded by the native substrate.
Hand-translate the two compute benches into the Scheme a jolt->Chez backend
would emit, to localize the execution-substrate ceiling without porting the RT.
fib 30: 246.6 -> 5.2 ms (~47x, fixnum). mandelbrot 200: 166.3 -> 13.4 ms
(~12.4x) ONLY with flonum-specialized ops; generic float ops box every flonum
and stay ~1.7x. 13.4 ms matches jolt's JOLT_CGEN C result, so Chez's native
compiler reaches the C ceiling with no cc step, REPL intact.
Size: Chez base 2.9 MB (AOT) / 4.0 MB (dynamic) vs Janet 2.21. Memory: Chez
~32-49 MB fixed baseline vs Janet ~12 MB (the one regression). RT-bound axes
(collections/binary-trees, where Chez's generational GC should help) not yet
measured. See spike/chez/RESULTS.md.
Generic language fixes so a host-shim library (jolt-lang/http-client) can carry
clj-http-lite without baking any HTTP/native code into core:
- reader: accept #^ (deprecated metadata reader macro) as ^
- (str pattern) returns the raw regex source, not #"..." — pattern composition
via (re-pattern (str p ...)) now works
- into/conj onto a map merges map items (was (into {} [{:a 1}]) -> {nil nil})
- a Var is callable as its current value (e.g. (wrap #'f) threading a var client)
- core-class reads a :class field off a thrown table, so libraries can throw
typed exceptions whose (class e) is a JVM class name
- compiled catch unwraps the interpreter's :jolt/exception envelope (__unwrap-ex),
so a typed throw keeps its class/message across the interpret/compile boundary
- slurp accepts a byte-array; io/copy is generic over :jolt/input-stream /
:jolt/output-stream stream markers
- instance? gains a registry (__register-instance-check!) for library shim types
- new clojure.test namespace (deftest/is/are/testing/use-fixtures/run-tests with
class-aware thrown?/thrown-with-msg?)
Spec: test/spec/clojure-interop-fixes-spec.janet.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Fix four bugs surfaced by the commonmark-app example
- regex: bounded quantifiers {n,m} no longer expand exponentially. The
desugaring inlined the continuation per optional level, doubling it each
time, so {0,61} built a PEG the compiler expanded to ~2^61 nodes and hung.
Each level is now its own grammar rule referenced by name (jolt-3xur).
- strings are a seqable of chars for vec/set/into, matching seq. They went
through realize-for-iteration, which had no string case, so into/set got
raw bytes (code points) and set threw; vec used string/from-bytes (1-char
strings). realize-for-iteration now maps make-char over the bytes, and
core-vec matches (jolt-dl4s).
- clojure.string/split takes Java Pattern.split limit semantics: negative
keeps trailing empties, 0/omitted drops them (a no-match result stays
[input]), positive caps the part count with the remainder unsplit. It used
to delegate the limit to take, so a negative limit returned [] and the
2-arg form never trimmed trailing empties (jolt-ik3a).
- System/exit is registered (maps to os/exit), so (System/exit n) works
(jolt-w2wf).
* regex: single-digit backreferences \1..\9 (jolt-xtss)
\1..\9 now match the text captured by the corresponding group, so
patterns like ([-*_])\1\1 or (\w+) \1 work. The parser records which
groups are referenced; a referenced group additionally captures its text
under a tag and the backref compiles to a PEG (backmatch). Only referenced
groups change — they match possessively (the CPS-over-possessive-PEG engine
can't backtrack into a tagged capture), so backtracking back into a
backreferenced group isn't supported (rare). Unreferenced groups keep full
backtracking and position-based result capture unchanged.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
canon-key canonicalizes a collection key by re-keying a native Janet
table by the canonical form of each element/entry. canon-key returns nil
for nil, and a Janet struct can't hold a nil key or value, so a nil set
member / nil map key / nil map value was dropped during canonicalization
— #{nil 1} canonicalized like #{1} and collided as a map key. So
(count {#{nil 1} :a, #{1} :b}) was 1 and (get {#{nil 1} :a} #{1}) was :a.
Box a nested nil before it goes into the table. The marker has to be
value-hashable, not the identity-hashed mutable-table sentinel the
transients use: the canonical struct becomes a long-lived phm key, and
its hash must survive the marshal/snapshot/fork that init-cached relies
on — an unmarshalled table gets a fresh address, so its hash isn't
preserved and the map can't find its own key. An interned keyword hashes
by content. Collision risk is only a real value equal to that exact
keyword, the same negligible class as canon-key's existing set/map struct
aliasing.
The transient sentinel stays a mutable table (it's built and consumed
within one op, never crossing a marshal boundary, so identity hashing is
stable there).
Co-authored-by: Yogthos <yogthos@gmail.com>
Two paths dropped a nil set member. phs-seq read members via
phm-to-struct, whose Janet struct can't hold a nil key, so the nil was
lost on seq/print and on into-an-existing-set even though the backing phm
counted it (count/contains? then disagreed). Re-attach the nil from the
phm's has-nil slot, keeping struct-key order for the rest so set printing
stays stable.
The transient set keyed its native table by canon-key and stored the
member as the value; canon-key returns nil for nil and a nil value also
drops the entry, so conj!/disj!/contains?/persistent! lost a nil member
outright. Key by tbl-key (the same nil sentinel the transient map uses)
and box a nil value through tbl-nil-key, unboxing on persistent!.
A nil-containing set used as a map key still collides with the nil-free
one (canon-key drops nil during key canonicalization) — separate latent
bug, filed as jolt-zcm9.
Co-authored-by: Yogthos <yogthos@gmail.com>
The map build already used a transient map, but each bucket was rebuilt with
a persistent (conj (get ret k []) x) per element — an O(log n) trie path
rebuild + alloc each. A coarse grouping (few large buckets) was bound on that
conj, not the map build. Buckets are now native arrays (transient vectors,
O(1) push) frozen once; distinct keys are tracked in a side vector so the
buckets freeze in place with no second map rebuild. A bucket's first element
stays a cheap persistent [x] and only promotes to a transient on the second,
so an all-singletons grouping pays no transient alloc.
coarse (10/100 buckets, 50k): ~313ms -> ~125ms (~2.5x)
2 buckets (50k): ~322ms -> ~129ms (~2.5x)
all-unique (50k): ~949ms -> ~892ms (no regression)
Surfaced a latent bug: canon-key returns nil for a nil key and Janet tables
drop a nil key, so the canon-keyed transient map silently lost a nil-key
entry — group-by/frequencies/assoc!/into{} dropped the whole nil bucket
((group-by identity [nil nil 1]) gave {1 [1]}, not {nil [nil nil], 1 [1]}).
Route nil through a sentinel (tbl-key) at the transient-map keying sites;
persistent!/count/dissoc! work unchanged since the real [nil v] pair is kept
as the stored value, and phm already has its own has-nil slot. The transient
set has the analogous bug (needs phs nil support) — filed separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
into {}, frequencies, group-by, set, into #{} and persistent! all built
their result by folding an immutable assoc/conj per element — each call
rebuilt the O(log32 n) trie path and allocated a fresh wrapper. Add a
one-pass bottom-up HAMT builder (phm-from-pairs) and route the builders
through it, the map/set analog of the pvec bulk build in #153.
phm-from-pairs partitions entries by hash and constructs the bin/array/
collision nodes directly, with the same bin<=16 / array-node>=17 promotion
the incremental path uses — so the trie is byte-identical to one built by
phm-assoc (validated across the size and branching boundaries, including
hash collisions, duplicate keys and the nil key). persistent! map/set and
the set constructor bulk-build; into {} keeps the small-scalar-map-stays-a
-struct rule via bulk-map-from-pairs; frequencies/group-by switch to the
canonical transient form and ride the fast persistent!.
50k A/B: into {} 704->270ms, frequencies 582->160, set 615->241,
into #{} 702->240, group-by 1358->919 (bound on persistent vector conj).
Gate: conformance x3, full suite (4718 >= baseline), new maps/sets bulk
boundary specs.
Co-authored-by: Yogthos <yogthos@gmail.com>
pv-from-indexed (behind vec/mapv/filterv/into-a-vector) built a pvec by calling
pv-conj once per element — each call allocated a fresh pvec wrapper and copied
the up-to-32 tail tuple, so building an n-vector was O(n) allocations + tail
copies. Replace it with a single bottom-up trie construction: chunk the elements
into 32-wide value leaves, group nodes 32-wide up to the root, split off the
tail. The structure is identical to the incremental one — tail-offset(n) =
((n-1)>>5)<<5 is exactly the trie/tail boundary, so nth/conj/assoc/seq read it
unchanged (validated against the old builder across the size boundaries).
into-a-vector likewise stops doing a persistent pv-conj per element: it
accumulates into a native array and bulk-builds once (the transient-style path).
Measured (50k): vec 211 -> 6 ms (~36x), into [] 197 -> 15 ms (~13x). mapv is
unchanged here — it's bottlenecked on lazy map realization, not the build.
The map/set builders (into {}, frequencies, group-by, set — all HAMT-backed)
need the same bulk treatment and are a separate follow-up. Gate: conformance x3,
full suite, new bulk-boundary rows in vectors-spec.
Co-authored-by: Yogthos <yogthos@gmail.com>
Add a PNG writer so the demos produce actual images. Two pieces:
- src/jolt/png.janet — the encoder (8-bit RGB, filter None, stored/uncompressed
DEFLATE so no compressor is needed; correct CRC32 + Adler32). It lives in Janet
because per-byte work in the overlay is far too slow (a byte-array aset loop is
~30s for 360k bytes, and CRC32 over even a tiny image would be worse). Janet's
bit ops are 32-bit signed, so the 32-bit-unsigned arithmetic is done with plain
number ops (doubles hold 2^32) plus byte-level bxor. Exposed to the overlay as
janet.png/* by importing it into eval_base's module-load-env.
- src/jolt/jolt/png.clj — the jolt.png overlay wrapper: image / put! / write. The
overlay only produces pixels; the host encodes them in one pass.
mandelbrot gets a `render` subcommand (jolt -m mandelbrot render out.png [size])
that colours count-point's escape counts; the numeric-arg bench path is untouched.
Verified end to end: macOS `sips` accepts the output (so CRC/zlib are valid).
png-test covers the encoder structure (signature/IHDR/IEND) and the overlay
round-trip.
Co-authored-by: Yogthos <yogthos@gmail.com>
emit-loop compiled every loop/recur to a self-recursive local closure called once
per iteration — relying on Janet TCO for stack safety but paying a fn frame + arg
bind each iteration. The jolt-5vsp spike localized the whole ~1.43x
jolt-over-hand-Janet gap on compute loops to exactly this.
Lower instead to a Janet `while` + state vars: the loop bindings become vars
carried across iterations, a recur writes them and raises a continue flag, and a
non-recur tail value falls out through a result var. recur-name routing in
emit-recur picks the while-set lowering for loops and leaves the fn-arity self-call
path untouched.
The one subtlety is closure capture: Janet closures capture vars BY REFERENCE, so
a closure built in the body over a shared mutable loop var would see the final
value ([3 3 3]) instead of its iteration's ([0 1 2]). Each iteration rebinds the
loop names into a fresh immutable `let` before running the body, which restores
per-iteration capture. recur reads those immutable bindings and writes the state
vars, so cross-referencing args (swap, fib) need no temps.
mandelbrot 218 -> 164 ms (~11.2x JVM, from 15x). fib is unaffected — it's
fn-arity recursion, not a loop. Regression spec in control-flow-spec covers
closure capture, no-clobber recur, nested loops, sequential init, recur through
let, and that fn-arity recur still works. Gate green (conformance x3, full suite).
Note: validating this after a rebuild needs JOLT_NO_DEPS_CACHE=1 — the deps-image
cache keys on the version string, not build identity, so it served stale codegen
(filed separately).
Co-authored-by: Yogthos <yogthos@gmail.com>
Fuse an app's native-compiled numeric-leaf fns plus its source into one
static executable: no sidecar .so, no toolchain on the target. The AOT path
(#148) already produced a prebuilt module + manifest; this links them into
the jpm-built exe so the app ships as a single file.
`jolt cgen-build -m NS -o OUT` stages a build dir (src/jolt-core symlinks
into the jolt tree, a generated cg.c of the hot fns, an uberscript bundle of
the app, and an entry that bakes the runtime, installs the native fns as var
roots, and runs -main), then runs `jpm build` there — declare-native builds
cg.a and declare-executable static-links it (jpm's create-executable marshals
the module cfns and calls its static entry at startup).
Build needs cc + jpm; the result needs neither. Mechanics that bit, codified
in cgen_build.janet: stdlib_embed slurps .clj cwd-relative so the build runs
in a repo-mirroring dir; jpm hardcodes ./project.janet and sets syspath=modpath;
the executable's dofile imports cg and static-links cg.a, neither ordered nor
release-built by default, so deps are wired explicitly; cleanup must lstat (the
tree symlinks must not be followed); the inner build runs --workers=1 so it
doesn't starve siblings in the parallel gate.
test/integration/cgen-build-test.janet builds the mandelbrot fixture, runs it
from a clean dir with no src/ and no cg.so, and checks the total at native
speed. Closes jolt-a7ds.
Co-authored-by: Yogthos <yogthos@gmail.com>
Lever 1 (native codegen) is built and merged (PRs #143-148): the floor is
localized, cgen translates numeric-leaf fns to C (JOLT_CGEN, 18x on mandelbrot,
cached), and the build-time AOT path deploys native code with no cc. Replaces the
stale START HERE (which still pointed at the now-done spike) with current status
and the open work: jolt-a7ds binary fusion, jolt-v28u while-lowering, jolt-l1l4
grammar widening, jolt-qx70 hot-fn detection.
Co-authored-by: Yogthos <yogthos@gmail.com>
Splits native codegen into a build phase (needs cc) and a deploy phase (none):
- gen-c-module/compile-module compile MANY numeric-leaf fns into ONE native
module (the AOT shape), generalizing the one-fn-per-.so JIT path.
- Backend :cgen-collect? records each numeric-leaf defn's IR while the app loads
as bytecode; cgen/aot-build compiles them into one module and write-manifest
persists {sopath, [{ns name sym}]}.
- Backend :cgen-prebuilt + cgen/load-aot: the deploy run loads the prebuilt .so
(via the native builtin, no cc) and installs each cfunction as the var root
with the same timing as the JIT path, so callers direct-link to native code.
- toolchain-available? no longer crashes when cc is off PATH (os/execute raises
on a missing exe) — a toolchain-less target now gets false.
Proven end-to-end in two processes (spike/native/aot-demo.janet): build with cc,
then deploy with cc removed from PATH -> count-point still native, mandelbrot
3288753 at 12.4ms (full 18x). Test: test/integration/cgen-aot-test.janet. Default
path unchanged; the modes are opt-in. Gate green (118 files).
Remaining for a literal single binary: fuse the .so + manifest into the jpm exe.
Co-authored-by: Yogthos <yogthos@gmail.com>
compile-fn now keys the .so on a hash of the generated C + the Janet ABI + the
platform, in a persistent cache dir (default jolt-cgen under TMPDIR, override
with JOLT_CGEN_CACHE_DIR; JOLT_CGEN_NO_CACHE=1 forces a rebuild). cc runs only on
the first build of a given fn; later runs with the same source reuse the cached
.so, so the per-startup compile cost is paid once.
mandelbrot 100 whole-process wall: cold ~0.71s -> warm ~0.21s (the ~0.5s cc
cost). These cache knobs don't shape output, so they stay out of
ctx-shaping-env-vars (same as the image-cache knobs). Test asserts the .so is
content-addressed and a second compile hits the cache without the source .c.
Co-authored-by: Yogthos <yogthos@gmail.com>
Wires src/jolt/cgen.janet into the backend's :def emit. With JOLT_CGEN=1 (off by
default, needs direct-linking), a plain defn of a numeric-leaf fn is compiled to
C at def time and the cfunction installed as the var root, so direct-linked
callers embed native code. The fn is not inline-stashed when cgen fires —
callers must call the C fn, not inline the bytecode body. ^:redef/^:dynamic stay
bytecode.
The leaf-first rule falls out: run calls count-point (a user var), so run isn't a
numeric leaf and stays bytecode, calling the native count-point over the cheap
forward crossing. mandelbrot 200: 224ms -> 12.4ms (~18x), result unchanged.
Adds JOLT_CGEN to ctx-shaping-env-vars (rides the disk-cache key) and :cgen? to
resolve-run-mode. Default path (cgen off) is a no-op: cgen-root returns nil and
the normal bytecode emit runs. Gate green (117 files). Test:
test/integration/cgen-pipeline-test.janet.
Co-authored-by: Yogthos <yogthos@gmail.com>
First slice of the native-codegen tier. A new standalone module, src/jolt/
cgen.janet, that translates a numeric-leaf fn (numeric in/out, body uses only
native-op arithmetic + loop/recur/if/let/do) to a Janet native C module: params
unboxed to C doubles at entry, loop/recur lowered to a while loop, reboxed at
return. compile-fn runs cc and loads the .so via the native builtin, returning a
cfunction; it returns nil for non-candidates or when the toolchain is absent.
count-point compiles and matches the bytecode fn across the mandelbrot grid
(test/integration/cgen-test.janet, which skips the behavioral leg where cc/janet.h
are missing). Nothing wires this into the default compile path yet — detecting
hot fns and installing the C version onto the var cell is the next step.
See docs/foundational-runtime-lever1-native-codegen.md for the ceiling
(native-C ~18-22x faster than bytecode, edges out JVM) and the leaf-first rule.
Co-authored-by: Yogthos <yogthos@gmail.com>
Probes the ceiling and incremental strategy for compiling hot fns to native C,
the only lever that moves the ~10.8x Janet-VM floor the localization spike found.
Native-C mandelbrot (Janet native module) runs ~10-12ms — faster than JVM
Clojure (14.2ms) and ~18-22x faster than jolt's 219ms. The boundary cost is
asymmetric: a bytecode loop calling a C hot-fn 40k times is nearly free (~11ms),
but a C fn calling back into bytecode via janet_call costs ~3.5us/call (~152ms,
no win). So the strategy is leaf-first / whole-hot-cluster compilation, crossing
only at cold edges. A plain cc-built .so (no jpm) loads at runtime via require at
full speed, so the native tier fits jolt's dynamic compile model.
Adds the spike artifacts under spike/native/ and the writeup. Next step is
jolt-ihdp (IR->C for the numeric subset). No source changes.
Co-authored-by: Yogthos <yogthos@gmail.com>
The jolt-vs-hand-Janet-vs-JVM mandelbrot comparison splits the 15.4x floor
into two layers: a Janet-VM floor (~10.8x JVM, optimal while-loop Janet over
unboxed doubles — only native codegen moves it) plus a ~1.43x jolt loop-
lowering overhead on top. The overhead is entirely the loop/recur -> recursive-
closure-called-per-iteration lowering; hand-Janet written the same way matches
jolt, while a while+var/set version is 1.43x faster. So a cheap backend win
(jolt-v28u) sits above the structural native-codegen lever.
Adds the spike artifacts under bench/ and the results writeup; marks the spike
done in the handoff. No source changes.
Co-authored-by: Yogthos <yogthos@gmail.com>
The targeted-specialization work (jolt-ffn) concluded that the constant-factor
gap vs JVM is structural, not per-form: three targeted passes (field-read,
inline cache, ctor descriptor-bake) all came back flat. mandelbrot (pure
compute) is ~15x off JVM and that's the floor — Janet bytecode VM + mark-sweep
GC + indirect calls.
This doc hands off the successor epic (jolt-5vsp): the foundational levers
(native codegen, GC-pressure reduction, deeper devirt+inline) and, importantly,
the spike to run first — localize the 15x floor by comparing jolt-compiled vs
hand-written-Janet vs JVM mandelbrot before committing to any big lever. Also
records what not to repeat.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method reads its fields through the generic guarded keyword lookup
because the method's `this` param is untyped. defrecord now hints `this` with
the record type, the per-form inference seeds ^Record-hinted params (the
:fn branch previously typed all params :any — only the whole-program path
seeded phints), and run-passes feeds the inference the record shapes. So a
hinted param's field reads bare-index instead of going through the :jolt/type
tag guard.
This needed a with-meta fix: (with-meta sym ..) returned a proto'd table, so
symbol? was false and the macro-attached hint broke fn destructuring. Symbols
now carry metadata in-place in their struct (matching how the reader attaches
^hint), keeping symbol? true, as in Clojure.
Modest on dispatch (~3-5%): the field read is a small fraction of a dispatch;
the machinery (record-tag + protocol lookup + wrapper) dominates, which is the
inline-cache target (jolt-ez5h). But it's a correctness fix and lets any
^Record-hinted code — not just methods — drop the field-read guard per-form,
not only under whole-program.
Co-authored-by: Yogthos <yogthos@gmail.com>
The ray tracer is compute-bound and the three existing benches only cover
alloc / megamorphic-dispatch / collections. Add three axes the epic needs to
judge itself holistically:
- mono-dispatch: monomorphic protocol dispatch. Its jolt/JVM ratio (~110x) is
*worse* than megamorphic (~76x) — the JVM inline-caches a runtime-monomorphic
call site to near-free while jolt does a full registry dispatch (devirt only
fires on statically-proven receivers). Points at the call-site inline cache.
- mandelbrot: pure float compute, no alloc/dispatch. The floor at ~15x — native
arith already gets close to the JVM.
- fib: recursion, call + integer-arith overhead.
run.sh gains JVM=1, which runs each bench on JVM Clojure too and prints the
jolt/JVM ratio. collections sized up now that the map is a HAMT (jolt-684u).
README documents the axes and the current scorecard.
Co-authored-by: Yogthos <yogthos@gmail.com>
run-tests.janet runs the same file set as `jpm test` across a pool of worker
processes (one `janet FILE` each, ev-based). The full gate goes from ~790s
serial to ~98s here (8x), and more on CI where the heavy files don't thrash on
swap. CI and the docs point at it; `jpm test` still works serially.
Three things dominated the wall:
- Nine integration tests cold-built a compile ctx (~8s each); switch them to
api/init-cached so they share the prebuilt image. The cache key already
fingerprints the ctx-shaping env vars, so the direct-link ones share one DL
image and the rest share the plain one.
- core-bench's main ran on every gate (~35s of benchmark loops that assert
nothing); gate it behind JOLT_BENCH=1.
- cli-test spawned `janet src/jolt/main.janet` ~20 times at ~8s cold each
(340s under parallel load, and it was the whole wall); prefer build/jolt
(~20ms baked ctx) when present, fall back to from-source for an unbuilt tree.
type-check-test stays on cold init: a snapshot-loaded ctx loses the success
checker's op/msg detail (jolt-vley). jolt-pria tracks caching from-source
startup generally, which would let cli-test drop the build/jolt preference.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Bind *command-line-args* after the deps-image cache swap (jolt-4mui)
Under whole-program (deps-image cache active), `jolt -m NS ARG` dropped ARG:
run-main set *command-line-args* on the current ctx, but a cache HIT then
replaced ctx with the saved image (via `set ctx cached`), whose *command-line-
args* was whatever got baked when the image was saved. The stale binding won at
`(apply NS/-main *command-line-args*)`, so -main ran with the wrong (usually
default) args — silently, for any optimized -m program.
Move set-command-line-args to AFTER the cache swap so it binds on the final ctx.
Repro/regression in deps-cache-args-test.janet: first run builds the image
(arg "first"), second run (cache hit) must echo "second", not the baked "first".
* docs: RFC 0003 — phm is a HAMT, sorted colls a red-black tree
The transients RFC described phm as "bucket-based copy-on-write" and mused about
"if it ever becomes a HAMT" — it is one now (jolt-684u), and sorted maps/sets are
a red-black tree (jolt-0hbr). Update the deviation/future-work notes accordingly.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
