The 6s per-file deadline + baseline=3981 were tuned to a fast dev machine; a
slower CI runner could time out a sub-second finite file, dropping total-pass
below baseline and flaking CI red. The deadline is now an env var (default 6
locally); CI sets it to 20s for headroom.
Characterized the 6 timeouts: identical pass/timeout counts at 6s and 20s, so all
6 are genuinely-infinite hangs (killed at any deadline) and the 227 finite files
finish well under 6s. So no baseline margin is needed — a generous CI deadline
removes the flake risk while preserving full regression sensitivity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cross-dialect clojure-test-suite (jank-lang fork) is now a git submodule at
vendor/clojure-test-suite instead of a personal ~/src checkout. The harness reads
it from the vendored path only (skips cleanly if the submodule isn't initialized:
`git submodule update --init`).
CI already checks out submodules recursively (for vendor/sci), so the suite is now
fetched and — since `jpm test` recurses through test/ — the baseline (3981 pass /
66 clean) is enforced on every push/PR. Updated the checkout comment accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Running clojure-test-suite surfaced an Option A regression (3971 -> 3957),
isolated to two root causes, both around lazy seqs:
1. nil first element wrongly read as end-of-seq. core-empty?, core-seq, and
core-reverse tested a lazy seq's emptiness with (nil? (ls-first coll)) — but a
lazy element may legitimately be nil. With Option A's lazy `drop`, the `case`
macro's (empty? (drop 2 clauses)) hit a nil-first lazy seq at the `nil`
case-constant and collapsed the rest of the case (incl :default) to nil —
breaking 14 case.cljc assertions. Now they realize one cell (seq-done?-style)
instead of trusting ls-first.
2. lazy transformer over a non-seqable silently yielded empty. The eager path
threw (realize-for-iteration on a char/number errors); Option A's lazy-from
returned nil, so (first (remove nil? \a)) gave nil where Clojure throws.
lazy-from now rejects non-seqable scalars (number/boolean/keyword/char/symbol)
with "Don't know how to create ISeq from: …".
Result: suite 3971 -> 3981 pass (net gain), clean files 45 -> 66 (Option A makes
seq?/vector? match Clojure across many cross-dialect files). Baseline raised.
Gate: conformance 258x3 (+5 regression guards), lazy-infinite 44/44, suite
3981/66, fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(partition-all n) and (partition-by f) now return stateful transducers, so they
compose with into/sequence/transduce/comp (previously errored "called with 1
argument, expected 2"). Both buffer a window, emit on boundary, flush a partial
trailing window in the completion arity, and honor `reduced` (early stop drops
the in-progress window, no extra trailing partition) — matching Clojure.
partition-all: td-partition-all in core.janet (Janet, alongside the other td-*),
core-partition-all made variadic. partition-by: transducer arity added to the
overlay defn in Clojure with volatiles (stays with its lazy collection arity).
Gate: conformance 253x3 (+4 transducer cases incl comp + reduced), lazy-infinite
44/44, fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(sequence xform coll) was eager (transduce into a tuple), so it hung on infinite
input. Now it's a lazy buffered transducer pump: pulls one source element at a
time, pushes it through the transducer's reducing fn into a buffer, and emits
buffered outputs lazily — so (take 3 (sequence (map inc) (range))) works.
Honors `reduced` (early stop) and runs the completion arity to flush stateful
transducers. Normalizes the source via core-seq (walking a raw pvec/set would
misfire — seq-done? uses length, which counts a pvec table's keys).
The other jolt-b56 items stay native by design: sequential?/seqable? are
representation-coupled and representation-mode-sensitive (jolt-1vx); realized?
reads the tagged :realized flag; cat/eduction/transduce/halt-when/unreduced/
ensure-reduced are the transducer/Reduced kernel the issue says to keep native.
Note: partition-all has no transducer (1-arg) arity, so (sequence (partition-all
2) …) errors — a pre-existing gap, unrelated to this change.
Gate: conformance 249x3, lazy-infinite 44/44, fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audited all 43 realize-for-iteration call sites. Result: every? was the one
transformer-style leak — it realized the whole coll before iterating, so
(every? pos? (range)) hung instead of short-circuiting on the first false
(Clojure short-circuits). Now walks lazy input cell-by-cell via realize-ls and
stops at the first false; not-every? inherits the fix.
Everything else is a legitimate realization boundary (reverse/sort/into/vec/set/
list/str-join/doall/apply/transient/to-array/hash-*/shuffle/rand-nth — must see
all elements) or forces only finite/concrete input (drop-while's eager branch,
cycle's seed, nth's concrete branch). some is already lazy (overlay). realized?
reads the :realized flag without forcing. first/nth/take pull only as far as
needed.
Regression: deadlined (every?/not-every? pos? (range)) cases — would hang if the
leak returned.
Gate: conformance 249x3, lazy-infinite 42/42, fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause: lazy-seq/lazy-cat were defined in 30-macros, which loads AFTER the
seq/coll tiers (10-seq, 20-coll) that use them. In compile mode a tier's forms
are compiled as the tier loads, so (lazy-seq …) in those tiers was compiled when
lazy-seq was not yet a registered macro — i.e. as a CALL to the macro-as-function,
which at runtime returns its own expansion `(make-lazy-seq (fn* [] …))` as data.
That leaked form then flowed into ops like `odd?` (partition-by) → type errors,
or silently produced wrong structure. Interpret/self-host masked it (expand at
call time); the eager fallbacks and the earlier letfn versions masked it by
falling back to the interpreter.
Fix: define lazy-seq/lazy-cat in 00-syntax (loaded first), exactly as when-let
already is for the same reason. They use only seed fns (make-lazy-seq/coll->cells/
concat) + map. With the macro registered early, the seq/coll tiers compile
(lazy-seq …) correctly.
With the root fixed, interleave/reductions/tree-seq drop their letfn workarounds
and use the canonical recursive Clojure forms (top-level / fn-self-name recursion
inside lazy-seq), verified leak-free in compile mode with strict probes.
Regression guards added: partition-by with odd? (the strict pred that exposed the
leak; the prior case used identity which masked it), reductions over an infinite
range, tree-seq summed through a strict filter — all ×3 modes.
Gate: conformance 249x3, lazy-infinite 40/40, fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These three were the last eager transformers, blocked by jolt-r81: a self-
recursive lazy-seq in the overlay leaks its macro expansion under :compile? when
recursion goes through a top-level name or (fn name …) self-name. Rewriting the
recursion as letfn-bound (the form partition-by/mapcat/dedupe already use, which
compiles cleanly) sidesteps the bug. All three are now lazy in interpret,
compile, and self-host — completing Option A for every transformer.
interleave: canonical lazy cons-recursion (2-arity) + map/concat (n-arity).
reductions: letfn step accumulator. tree-seq: letfn walk + lazy mapcat.
Gate: conformance 246x3, lazy-infinite 40/40 (+interleave/reductions/tree-seq
infinite cases), fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Option A makes `take` lazy, so the §6.3 counter tests must force the take
result (dorun) to drive realization — an unconsumed take correctly realizes
nothing. With that, all 37 lazy-infinite cases pass and the minimal-realization
counts match (proving the Option A transformers realize exactly the demanded
prefix).
interleave kept eager: a lazy (cons-recursive) version leaks its lazy-seq macro
expansion under :compile? — the same jolt-r81 bug that blocks lazy reductions/
tree-seq. Documented in 20-coll.clj.
Gate: conformance 246x3, lazy-infinite 37/37, fixpoint, self-host, specs+unit
green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Converts the rest of the lazy transformers to return a LazySeq even over a
concrete collection (dropping the eager "preserve representation" branch, which
returned a vector over vector input): drop, distinct, partition, partition-all,
map-indexed, keep, keep-indexed, take-nth, interpose. Each collapses to its
existing lazy branch run over (lazy-from coll); transducer arities unchanged.
Fixes a latent nth bug exposed by this: core-nth's lazy branch walked with
(ls-first cur) truthiness as the end-of-seq test, so a legitimate false/nil
element was mistaken for the end — (nth (map identity [false 1 2]) 0) threw
instead of returning false, and cond-> (whose macro nths over a now-lazy
(drop 2 clauses) containing the boolean clause tests) failed. nth now walks
with seq-done? and reads via core-first.
Gate: conformance 246x3 (+7 cases), lazy-infinite 18/18, fixpoint, self-host,
all specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lazy transformers now return a LazySeq even over a concrete vector, matching
Clojure: (seq? (map inc [1 2 3])) is true, (vector? ...) false. Replaces the
"preserve representation" eager branch (which returned a vector over vector
input) by routing concrete colls through lazy-from + the lazy step machinery.
Flipping these surfaced four boundary bugs, all fixed here:
- cons over a lazy-seq returned a raw cell @[x thunk]; a cons-of-a-cons then
treated it as a plain 2-array and leaked the rest-thunk as an element (broke
interleave). cons over lazy now returns a proper LazySeq.
- coll->cells mistook a user vector whose 2nd elem is a function ([first last]
from juxt) for a cons cell. Cons cells are mutable arrays; user data is
immutable — route pvec/plist/tuple through immutable tuples and apply the
[val,fn] cell heuristic only to mutable arrays. Also coerce set/map/string/
buffer via core-seq.
- ~@ splice over a lazy map result iterated a LazySeq as a Janet table (broke
lazy-cat / self-ref fib). syntax-quote* now realizes via d-realize before
splicing; core-sqcat (self-host) already realized.
- core-next did (length r) on a lazy rest (never 0 on a table) and ls-rest
could return nil → (length nil) crash. core-rest never returns nil; core-next
uses seq-done? (realizes one cell). seq-done? moved above core-rest.
normalize-pvecs (test helper) realizes lazy-seqs so Janet-= comparisons work.
Gate: conformance 239x3 (interpret/compile/self-host, +10 Option A cases),
lazy-infinite 18/18, fixpoint, self-host, all specs+unit green. (sci-bootstrap
and clojure-test-suite skip — vendored dirs absent in this checkout.)
Remaining for full Option A consistency (jolt-7w4): drop/map-indexed/keep/
keep-indexed/take-nth/interpose/distinct/partition/partition-all still eager
over concrete input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Added 16 atom-counter laziness tests to lazy-infinite-test.janet:
map, filter, remove, take-while, drop-while, distinct, take-nth,
map-indexed, keep, keep-indexed, interpose, partition, partition-all,
mapcat, dedupe, repeated-inc
Each test wraps input elements in (swap! c inc) and proves only the
exact minimal number of elements are realized. 37/37 pass, 0 timeouts.
Updated phase-5.md §7: "22/22" → "21/21" (interleave commented out),
§6.3 marked Done.
Final gate results:
lazy-infinite: 37/37 (21 value + 16 counter)
conformance: 229/229 ×3
specs: 32/32 files, 0 failures
clojure-test-suite: 3971 pass (↑45), 6 timeouts (↓3)
core-bench: TOTAL 2531 ms (no Phase-4 baseline for comparison)
Root cause: lazy interleave in 20-coll.clj uses lazy-seq macro which
expands to (make-lazy-seq (fn* [] (coll->cells ...))) in compile mode.
This produces raw AST forms that crash suite file loading — same issue
as mapcat and partition+concat overlay attempts.
Fix: revert interleave to eager (vec-based loop). dedupe stays lazy
(uses make-lazy-seq directly, not lazy-seq macro). xml-seq stays
(uses tree-seq which is eager in overlay).
Suite: 3971 pass (up from 3926), 6 timeouts (down from 9), 4628 assertions.
Lazy-infinite: 21/21 (interleave infinite case commented out).
Conformance: 229x3.
Fix & rest destructuring (evaluator.janet): when the original
value is a lazy-seq, derive & rest by walking ls-rest vi steps
instead of slicing the eagerly-realized array from d-realize.
Rewrite core-mapcat as a lazy state machine (core.janet): step
through input colls one element at a time, call f on each tuple,
then yield from f's result collection. No apply-forcing —
all input colls are walked lazily via lazy-from + seq-done?.
Add mapcat to core-renames (compiler.janet) and remove from
Clojure overlay (10-seq.clj) — compile mode now emits direct
calls to the lazy core-mapcat, fixing the pre-existing
protocol-on-record compile error.
Tests: re-enabled mapcat infinite harness case, added 2 & rest
laziness cases. 21/21 lazy-infinite, 229x3 conformance, 32/32 specs.
core-drop-while (core.janet): return cell via realize-ls instead
of raw LazySeq. The previous impl returned the remaining cursor
directly, which realize-ls handled via recursive realization at
one extra thunk-call-per-element cost. Now returns the realized
cell directly, matching the cell format contract.
mapcat (10-seq.clj): revert to standard (apply concat (apply map
f colls)). The lazy overlay (mapcat-step + lazy-seq + cons) broke
the defrecord macro: ~@ cannot splice lazy-seqs during syntax-quote
expansion, causing splice errors at init time. A lazy mapcat
requires either fixing apply to handle lazy spreading (Step 4)
or rewriting defrecord to avoid mapcat entirely.
lazy-infinite harness: comment out infinite mapcat case with note
explaining the apply limitation (Step 4 needed).
Gates verified:
conformance 229/229 x3 (interpret/compile/self-host)
lazy-infinite 18/18
specs 32/32 files, 0 failures
mapcat in 10-seq.clj now uses a defn- mapcat-step helper that walks
lazy map results element-by-element (cons + lazy-seq), with explicit
arities for 1/2/3/4+ colls. The 4+-coll arity still uses apply to
spread colls to map, but apply no longer forces the inner map result.
Previously (apply concat (apply map f colls)) forced the lazy map
result through core-apply realize-for-iteration, which hung on
infinite inputs. The new step-based impl walks one element at a time
via first/rest/seq primitives.
Re-enabled deferred mapcat infinite-input harness case. 19/19 pass.
Gates: lazy-infinite 19/19, conformance 229/229 interpret+self-host
(compile 228/229 — 1 pre-existing protocol error), specs 32/32.
reduce-kv is pure over reduce/keys/get/nth, so it moves to 20-coll with no
host surface. Both branches fold through reduce, which fixes two latent bugs
in the Janet version: on a vector it saw the pvec as a table and folded over
its internal keys instead of the indices, and it ignored reduced entirely.
nil folds to init. Added vector-index, reduced, and nil spec cases plus a
3-mode conformance case for the vector fix.
Spec suites (interpret) for the overlay-migrated fns and both fixes:
- control-flow: if-let/when-let/if-some/when-some else-scope (9 cases)
- predicates: not-any?/idents/numeric preds/NaN?/abs/object?/... (24 cases)
- sequences: nthrest()/nthnext/distinct? value-eq/replace nil/take-last/... (23)
Plus 10 cases added to the all-3-modes conformance set for the highest-risk
fixes (if-let scope, nthrest (), distinct? value-eq), so compile + self-host are
guarded too: conformance 218 -> 228 in all three modes.
Full suite green.
Add a maps-spec defspec covering nil values through the reader (literals), the
construction path, and the op surface (assoc/merge/merge-with/into/conj/zipmap/
select-keys/get-in/dissoc/reduce-kv, nil keys). Raise the clojure-test-suite
baseline 3919 -> 3926 to guard the assertions the fix unlocked.
The reader yields set literals as a form ({:jolt/type :jolt/set ...}); edn now
walks the parsed result and builds actual sets (recursing into maps/vectors;
lists stay lists, never evaluated). Untagged-struct guard so symbols/chars/tagged
literals pass through. 47 -> 49 in the battery; only #uuid (no uuid type) remains.
clojure.edn/read-string now delegates to clojure.core/read-string (the
reader-based special form — parses, never evals) via a private helper, with the
opts-map :eof arity and nil/blank-input handling. 43 -> 47 in the stdlib battery.
This required an evaluator fix: special forms were matched by name only, so
clojure.edn/read-string dispatched the core read-string SPECIAL FORM instead of
the var. Now a head qualified to a non-core namespace resolves to its var; only
unqualified or clojure.core-qualified heads are special forms. Conformance
218/218 all modes; full suite green.
Remaining edn gaps (jolt-b7y): set literals read as a set FORM not a constructed
set, and #uuid has no real type.
clojure.data: ported from the ClojureScript impl (jolt-native equality-partition
fn instead of host-type protocol dispatch). Verified against Clojure's own
data-test (12/13 canonical; the 13th needs nil-valued-map support, jolt-c7h).
Used mapv over the reference's (doall (map …)) — jolt's lazy multi-coll map +
doall don't force as a reduce accumulator (jolt-dzh).
clojure.zip: jolt's was a broken custom reimplementation; replaced with a port of
real clojure.zip (metadata-based locs). Uses (nth loc …) instead of (loc …)
because meta-bearing vectors aren't invocable as fns (jolt-vh5).
core: with-meta now accepts nil metadata (Clojure allows (with-meta x nil)); it
crashed calling keys on nil. This is what unblocked zip's make-node.
New vendored battery (test/clojure-stdlib/, from clojurust's suite with fixtures
corrected to match real Clojure): walk 34, zip 33, data 61 all clean; edn guarded
at 43 (still a stub — jolt-b7y). Conformance 218/218 all modes; full suite green.
The expensive multimethod path is the hierarchy fallback: when a dispatch value
isn't a direct method key, mm-fn walks every key with isa? (derive-based
dispatch). That resolution is now cached per dispatch value on the multimethod
var (:jolt/dispatch-cache), cleared in place by defmethod/prefer-method/
remove-method/remove-all-methods so a redefinition can't be served stale. Direct
(get methods dv) hits stay uncached — already a single lookup.
Test cases added to dispatch-cache-test (compile, interpret, aot-core off):
hierarchy hit is cached, a newly-added specific method is seen, removal re-exposes
the fallback. Conformance 218/218; full suite green.
Soundness gate for self-hosting (jolt-d0r). stage1 = analyzer built by the Janet
bootstrap; stage2 = analyzer rebuilt by compiling jolt.ir + jolt.analyzer through
stage1 (self-host); stage3 = the same self-rebuild again. A corpus of programs
must produce identical results across all three.
Compared BEHAVIORALLY rather than by emitted code: emitted Janet forms embed live
setter/getter closures (identity varies per compile) and the IR carries
representation-level gensyms and pvec internals, so structural equality fights the
representation. Behavioral parity is the property that actually matters and is
representation-independent.
Also corrects the battery baseline 3920 -> 3919: 3920 was a lucky run; 3919 is the
stable value (the 9 timeout-prone tests can shift the count by one).
Three pure function combinators into the collection tier, as compositions of
mapv/every?/some/apply. Three more core-* primitives + table entries gone.
Battery 3916 -> 3920 (the canonical defs are more correct than the prior Janet
ones — some-fn returns the matching value, every-pred is properly boolean).
Conformance 218/218 all modes.
Host-value protocol dispatch (a protocol extended to Number/String/… called on a
host value) recomputed the candidate type-tag list and walked the registry — up
to ~15 map lookups plus an array alloc — on every call. It now caches
(most-specific-host-tag, protocol, method) -> fn, guarded by a registry
generation that bumps in register-protocol-method; re-extending a protocol
invalidates the cache. The single-lookup fast paths (deftype protocol dispatch,
direct multimethod dispatch) are untouched — they're already cheap, so no cache
overhead there. The multimethod hierarchy-fallback cache (narrower: only
derive-based dispatch) is deferred.
Test in test/integration/dispatch-cache-test.janet covers correctness under
re-extension in all three modes. Conformance 218/218; battery 3916.
Calls compile direct (target value embedded) iff the compiling unit has
direct-linking on, the target isn't ^:redef/^:dynamic, and it's an
already-defined fn; otherwise indirect (live var deref, redefinable). Direct
emission is guarded to function values only — embedding a non-function would make
Janet evaluate it as code.
:aot-core? (init opt, default true; JOLT_AOT_CORE=0 disables) compiles the core
tiers with direct-linking on so core->core calls are direct. User/REPL code
compiles indirect by default, so redefining functions in the REPL works with no
annotation, exactly like Clojure. :aot-core? false makes core indirect too — the
whole language becomes redefinable. ^:redef / ^:dynamic on a target force it
indirect even inside a direct-linked unit.
Also fixes a pre-existing gap this surfaced: compiled def dropped ALL var
metadata (so ^:dynamic was silently lost under compilation, and ^:redef couldn't
work). def metadata now flows analyzer -> IR -> back end (new form-sym-meta host
fn, def-node :meta, var-setter-meta) and is applied to the cell.
Matrix test in test/integration/direct-linking-test.janet. Conformance 218/218
in all three modes under both :aot-core? true and false; battery holds 3916;
binary rebuilt.
First fractal turn of the multi-stage bootstrap (jolt-tzo). The Clojure part of
clojure.core is now loaded in ordered tiers under jolt-core/clojure/core/. The
kernel tier (00-kernel: second/peek/subvec/mapv/update) holds the structural fns
the self-hosted compiler itself uses; in compile mode it is bootstrap-compiled
into clojure.core BEFORE the analyzer is built, so the analyzer binds those names
to the Clojure definitions instead of a not-yet-defined forward ref. That removes
the circularity that previously forced these to stay in Janet — the five core-*
primitives and their init-core! entries are gone.
Mechanism: backend/bootstrap-load-source (generalized from compile-load) builds a
source string into a target ns via the bootstrap; rebuild-compiler! recompiles
the self-hosted compiler against the current core (the rail for future turns,
exercised by the new fixpoint test). api/load-core-overlay! walks the ordered
tiers, bootstrap-loading kernel tiers and self-hosting the rest.
Also fixes a latent evaluator bug surfaced by the move: a fn rebinds current-ns
to its defining ns while it runs and restores on normal return, but a fn that
THREW unwound past its own restore, leaking the ns. try now restores the
try-entry ns on the catch/finally path, so referred-symbol resolution survives a
caught error (this was breaking is/testing in the suite harness after any thrown
assertion). Net battery +3 (3913 -> 3916); conformance 218/218 in all three
modes; binary builds and runs the embedded tiers.
Phase 4 kernel-shrink seam. jolt-core/clojure/core.clj holds the Clojure portion
of clojure.core; api/init loads it into the clojure.core namespace after
init-core! interns the Janet primitives, routed through eval-toplevel so it
compiles via the self-hosted pipeline (or interprets when :compile? is off).
First three fns moved off the Janet side: ffirst, fnext, nnext (leaf, no internal
callers). Same results compiled or interpreted; full suite green; battery holds
3913.
eval-toplevel now routes every compiled form through backend/compile-and-eval
(analyzer -> IR -> Janet back end). The bootstrap compiler (compile-ast) stays
only to bootstrap jolt.ir/jolt.analyzer in backend/compile-load. Dropped the
:selfhost? flag and JOLT_SELFHOST env plumbing — self-host is the default.
Three correctness fixes the full-suite-under-self-host sweep exposed:
- analyzer/back end: recur into a variadic arity now compiles. The arity fn
gets an ordinary positional rest param (the collected seq), with a dispatch
wrapper, so (recur fixed... rest-seq) is a clean self-call like Clojure --
instead of punting to the interpreter, which hangs on this (jolt-4df).
- host_iface: interop heads (.foo/.-field/Foo.) and ns-management forms
(all-ns, create-ns, resolve, ...) now mark uncompilable so they fall back
instead of compiling to a bad call.
- back end: named-fn self-reference ((fn self [n] ... (self ...))) binds the
fn name via a var.
Full unit+integration suite green; clojure-test-suite battery holds 3913.
conformance-test now runs every case under interpret, the bootstrap compiler, AND
the self-hosted pipeline (portable Clojure analyzer -> IR -> Janet back end). All
three pass 218/218, so the self-hosted compiler can't silently regress in CI.
suite-worker honors JOLT_SELFHOST=1 to route the clojure-test-suite through the
self-hosted pipeline (for validating once the analyzer compiles).
The analyzer stays interpreted: compiling it via the bootstrap miscompiles
(qualified refs regressed compile mode 3932->1181; compile-loading yields
corrupted IR) — tracked in jolt-4xc. Correctness is unaffected (the self-hosted
pipeline passes 218/218 conformance); only speed is pending.
Analyzer threads an env (locals + recur target) instead of a bare locals set, and
compiles loop*/recur, recur directly in a fixed-arity fn (each arity's name is its
recur target -> self-call), and try/catch/finally (-> Janet try + defer). recur
into a variadic arity isn't a recur target, so it falls back to the interpreter
rather than miscompiling the rest seq. Still 218/218 conformance via the pipeline,
now with these forms compiling natively instead of interpreting.
The Clojure-in-Clojure front end now compiles and runs a real subset end to end:
arithmetic, if/do/let, vector/map literals, def + name-based global refs, fn,
defn (via macroexpand), recursion through the var cell, multi-arity, variadic,
and higher-order calls. Proven by test/integration/self-host-test.janet.
Pieces:
- jolt-core/jolt/analyzer.clj — PORTABLE Clojure analyzer: reader form -> IR,
depending only on the host contract (jolt.host), never on Janet.
- src/jolt/host_iface.janet — Janet impl of the jolt.host contract (form
introspection + resolve/macro/current-ns), installed into each ctx.
- src/jolt/backend.janet — Janet back end: IR -> Janet form -> eval; resolves
name-based :var nodes to cells and reuses runtime helpers.
Host Janet code lives in src/jolt/ (not a host/janet/ dir): Janet resolves
relative imports per file, so cross-dir ../../src/jolt/* imports load second
instances of compiler/types/core and corrupt state. The portability boundary is
the jolt.host namespace contract + jolt-core/, not the directory.
Notes: gensym in the back end must not be Janet's (shadowed by Jolt's via use);
a jolt bug — (into #{} coll) doesn't add elements — was worked around with reduce
conj in the analyzer (filed separately).
Adds src/jolt/aot.janet: save-ns / load-ns-image marshal a namespace's compiled
var cells to a Janet bytecode image and load them back, skipping
parse/analyze/emit/compile on reload.
Compiled functions close over core fns (cfunctions) that can't marshal by value,
so we marshal against a dictionary built from the baked-in runtime env
(env-lookup jolt-runtime-env, which chains to the Janet boot env): core fns and
builtins are referenced by name, only user bytecode and its var cells are
serialized. The runtime env is identical at save and load time (same binary), so
the dictionaries match.
Test round-trips a namespace (constant, fn over it, core-fn user, recursion) into
a fresh context and confirms the loaded vars run and stay redefinable.
analyze-form now mirrors resolve-sym's full resolution order: jolt var (current
ns / clojure.core), then the runtime/Janet env, then a pending forward-ref cell.
Previously a name that resolves only via the Janet-env fallback (int?, type, …)
interned an empty var and miscompiled; now it emits the runtime binding directly,
matching the interpreter.
suite-worker honors JOLT_COMPILE=1 to run the whole clojure-test-suite through
the compile path. Under compilation it now passes 3932 / errors 124 — at parity
with the interpreter baseline (3913 / 125) across 4600+ assertions, confirming
the hybrid compile path doesn't diverge from the interpreter.
Running the conformance suite under compile mode surfaced many forms that
silently miscompiled (the hybrid fallback only catches compile-time errors,
not wrong results). Fixes:
- Global var resolution now mirrors the interpreter's resolve-var: current ns
(which holds refers) then clojure.core, instead of interning an empty var in
the current ns. This was the big one — every core fn not in core-renames
(iterate, update, subvec, reductions, ...) derefed to nil from a user ns.
- Map literals evaluate their keys and values (were emitted as quoted data);
build via build-map-literal, mirroring the interpreter (struct, or phm when a
key is a collection).
- Vector literals build a mode-appropriate jolt vector via make-vec (pvec when
immutable, array when mutable) instead of a bare Janet tuple, so compiled and
interpreted vectors share one representation (type-strict ops like rseq
rejected tuples).
- Core fn values resolve dynamically from the runtime env instead of a
hand-maintained table that had drifted: core-apply mapped to Janet's native
apply (rejects pvec tails), core-some mapped to core-some?. Removed the table
and the bogus some rename.
- analyze-form throws uncompilable on interpreter special forms it doesn't
implement and on definitional/host macros (deftype, defprotocol, reify,
binding, letfn, read-string, regex/tagged literals, ...), so they fall back
to the interpreter instead of miscompiling — including nested in compiled
forms.
conformance-test.janet now runs every case under both interpreter and compiler
so compile-mode correctness is guarded in CI.
Rewrites fn* analysis/emit. Each arity compiles to a named Janet fn whose
name is its recur target, so recur is a self-call (Janet tail-calls it) —
this fixes recur used directly inside a fn, which previously miscompiled to
a runtime error that the hybrid fallback couldn't catch.
Multi-arity fns dispatch on arg count: fixed arities match exactly, the
variadic arity matches >= its fixed count. The rest param is an ordinary
param holding a seq, so (recur fixed... rest-seq) into a variadic arity
works as in Clojure. Single fixed-arity fns keep the direct, dispatch-free
shape so the hot path stays fast.
Destructuring params still route to the interpreter via uncompilable.
Tests cover named recursion, multi-arity (incl. variadic clause), recur in
fn, and recur into a variadic arity.
eval-one now tries to compile each form and falls back to the interpreter
for anything the compiler can't handle, instead of erroring or silently
miscompiling. Only the compile step is guarded, so runtime errors in
compiled code still propagate (no double-eval of side effects, no hidden
errors).
The compiler now throws jolt/uncompilable on the forms it would otherwise
miscompile — destructuring let/loop bindings and fn params, multi-arity
fns, and named fns — so they route to the interpreter and produce correct
results. These become compile targets in a later optimization pass.
Adds eval-compiled to split compile from eval; compile-mode tests cover
destructuring, multi-arity, named-fn recursion, and error propagation.
Compiled global references now deref through the Jolt var cell instead of an
early-bound Janet symbol, so redefining a def/defn is visible to already-compiled
callers (Janet early-binds plain symbols). A global ref analyzes to :op :var
(resolving/creating the cell); a def sets that same cell's root. Because Janet
copies table constants but references functions, we embed memoized getter/setter
closures over the cell rather than the cell itself. This also subsumes the old
named-fn recursion rewrite and the separate ns-intern.
ns-intern now stores the namespace *name* (string) in a var, not the ns table, so
var cells aren't cyclic (required for embedding).
fib(30) compiled ~0.5s (the late-binding cost vs phase 2's direct-linked 0.08s;
direct-linking for hot calls is a later optimization). Suite green; cross-form,
recursion, and context isolation preserved; redefinition now works.
The Clojure stdlib namespaces (clojure.string/set/walk/edn/zip, jolt.http/
interop/shell/nrepl) were loaded cwd-relative, so the built binary couldn't
load them outside the repo. Now stdlib_embed slurps them all at build time into
a {ns -> source} map frozen in the image, and the loader falls back to it when a
namespace isn't found on disk. The artifact is self-contained — clojure.string
works from any directory.
Drops the one-off nREPL source embed (jolt.nrepl is covered by this now). The
baked-in stdlib is excluded from uberscript bundles since it's part of the
runtime. clojure.core is unchanged (Janet, auto-referred).
Test: embedded-stdlib-test requires from the embedded copy with FS roots cleared.
Phase 3 — bundling. `jolt uberscript OUT -m NS` requires NS, uses the loader's
recorded load order (a dep finishes loading before its requirer, so it's
topological) to concatenate the used namespaces into one .clj that runs on a
plain jolt with no deps/jpm. `jolt-deps uberscript` resolves first. The loader
records loaded file paths when ctx env has :loaded-files.
Phase 4 — conformance. deps-conformance-test resolves real pure-cljc git libs
and reports load/run status, gated behind JOLT_CONFORMANCE so CI stays offline.
First run: medley works; cuerdas hits a reader gap (filed); stuartsierra/
dependency uses Long/MAX_VALUE (JVM interop, out of scope).
Tests: uberscript-test, deps-conformance-test (skips without the flag).
Add --version/version, -e/--eval, -f/--file, -m/--main (require NS and apply
its -main to *command-line-args*), and nrepl-server/--nrepl-server taking an
optional [host:]port (nrepl stays an alias). repl/help/-h round it out.
Also rewrite doc/tools-deps.md as design notes for the now-implemented deps
support (it still read as pre-implementation research), and note the new flags
in the README. Test: cli-test runs the flags from source.
The deps tests built temp paths from $TMPDIR, which isn't set on the CI
runners — deps-resolve-test then tried to mkdir at the filesystem root and
failed. Default to /tmp.
A separate jolt-deps executable (mirroring jpm beside janet) resolves a
deps.edn into source roots and runs jolt with them on JOLT_PATH; the runtime
stays deps-agnostic.
- src/jolt/deps.janet: read deps.edn (Janet parses it directly), resolve
:git/* and :local/root via jpm/pm's download-bundle (git fetch + cache, no
build), recurse for transitive deps, return de-duped roots. jpm loaded
lazily so it's never embedded. Roots cached on a deps.edn hash.
- src/jolt/deps_cli.janet + project.janet: the jolt-deps tool — path/run/repl/-e.
- main: apply JOLT_PATH at runtime (the CLI ctx is frozen into the image at
build, so init's env read wouldn't see it).
Verified end to end against a local dep and a real git dep (medley). git only,
pure clj/cljc. Test: deps-resolve-test (local deps, no network).
The loader resolved a namespace against one hardcoded root (src/jolt, .clj
only). Now it searches an ordered list (ctx env :source-paths, stdlib first)
and tries .clj then .cljc. init adds roots from the :paths opt and JOLT_PATH.
This is the foundation for loading deps.edn libraries; verified loading medley
from an added root. No change to stdlib loading (3913 suite baseline held).
- core: pr-render/str-render now render a Jolt var as #'ns/name instead of
falling through to the generic table printer, which recursed into the var's
cyclic :meta/:ns refs and looped forever. Adds name-of/var-display helpers.
- jolt.nrepl: capture the post-eval namespace *after* running the form — jolt
evaluates map-literal values right-to-left, so {:val (eval form) :ns (the-ns)}
read the ns before in-ns ran, pinning every session to 'user'. Now in-ns/ns
switches persist across evals. render-value dropped: pr-str handles vars.
- specs: var printing (strings-spec) and nREPL in-ns/ns-persistence + explicit
:ns override (nrepl-test).
