These three were the last eager transformers, blocked by jolt-r81: a self-
recursive lazy-seq in the overlay leaks its macro expansion under :compile? when
recursion goes through a top-level name or (fn name …) self-name. Rewriting the
recursion as letfn-bound (the form partition-by/mapcat/dedupe already use, which
compiles cleanly) sidesteps the bug. All three are now lazy in interpret,
compile, and self-host — completing Option A for every transformer.
interleave: canonical lazy cons-recursion (2-arity) + map/concat (n-arity).
reductions: letfn step accumulator. tree-seq: letfn walk + lazy mapcat.
Gate: conformance 246x3, lazy-infinite 40/40 (+interleave/reductions/tree-seq
infinite cases), fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Option A makes `take` lazy, so the §6.3 counter tests must force the take
result (dorun) to drive realization — an unconsumed take correctly realizes
nothing. With that, all 37 lazy-infinite cases pass and the minimal-realization
counts match (proving the Option A transformers realize exactly the demanded
prefix).
interleave kept eager: a lazy (cons-recursive) version leaks its lazy-seq macro
expansion under :compile? — the same jolt-r81 bug that blocks lazy reductions/
tree-seq. Documented in 20-coll.clj.
Gate: conformance 246x3, lazy-infinite 37/37, fixpoint, self-host, specs+unit
green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause: lazy interleave in 20-coll.clj uses lazy-seq macro which
expands to (make-lazy-seq (fn* [] (coll->cells ...))) in compile mode.
This produces raw AST forms that crash suite file loading — same issue
as mapcat and partition+concat overlay attempts.
Fix: revert interleave to eager (vec-based loop). dedupe stays lazy
(uses make-lazy-seq directly, not lazy-seq macro). xml-seq stays
(uses tree-seq which is eager in overlay).
Suite: 3971 pass (up from 3926), 6 timeouts (down from 9), 4628 assertions.
Lazy-infinite: 21/21 (interleave infinite case commented out).
Conformance: 229x3.
1. partition+concat removed from 10-seq.clj overlay — they use
lazy-seq macro which produces raw make-lazy-seq forms in compile
mode (same issue as mapcat overlay reversal). Must stay native.
2. Dead defns removed from core.janet: core-map-indexed,
core-iterate, core-repeatedly (30 lines). Bindings already
removed in prior commit.
3. dedupe removed from 40-lazy.clj — belongs in 20-coll.clj.
Gates: conformance 229×3, lazy-infinite 22/22, specs 32/32.
The lazy-seq macro (30-macros.clj) is not available when 20-coll
loads. Use explicit (make-lazy-seq (fn* [] (coll->cells ...)))
expansion. Also fix outer cons leaking raw @[val thunk] cell.
22/22 lazy-infinite, conformance 229 interpret + self-host.
The git stash pop restored the old lazy-seq version of dedupe.
Re-applied the make-lazy-seq + coll->cells fix for the 20-coll
tier where lazy-seq macro is not yet available.
Overlay trampoline fails arity dispatch — (trampoline f 5) with [f]
and [f & args] arities dispatches incorrectly in Jolt. The Janet
core-trampoline is 4 lines and works correctly. Kept native.
Results: 22/22 lazy-infinite, 229x3 conformance, 32/32 specs.
dedupe: use make-lazy-seq + coll->cells directly — lazy-seq
macro is not available at 20-coll tier (defined in 30-macros.clj).
trampoline: use ifn? instead of fn? (not available), fix
single-arity to call (f) directly instead of recursive (trampoline f (f)).
rand-int: removed from overlay, stays native in core.janet
(math/floor and math/random not available in Clojure namespace).
Fix & rest destructuring (evaluator.janet): when the original
value is a lazy-seq, derive & rest by walking ls-rest vi steps
instead of slicing the eagerly-realized array from d-realize.
Rewrite core-mapcat as a lazy state machine (core.janet): step
through input colls one element at a time, call f on each tuple,
then yield from f's result collection. No apply-forcing —
all input colls are walked lazily via lazy-from + seq-done?.
Add mapcat to core-renames (compiler.janet) and remove from
Clojure overlay (10-seq.clj) — compile mode now emits direct
calls to the lazy core-mapcat, fixing the pre-existing
protocol-on-record compile error.
Tests: re-enabled mapcat infinite harness case, added 2 & rest
laziness cases. 21/21 lazy-infinite, 229x3 conformance, 32/32 specs.
core-drop-while (core.janet): return cell via realize-ls instead
of raw LazySeq. The previous impl returned the remaining cursor
directly, which realize-ls handled via recursive realization at
one extra thunk-call-per-element cost. Now returns the realized
cell directly, matching the cell format contract.
mapcat (10-seq.clj): revert to standard (apply concat (apply map
f colls)). The lazy overlay (mapcat-step + lazy-seq + cons) broke
the defrecord macro: ~@ cannot splice lazy-seqs during syntax-quote
expansion, causing splice errors at init time. A lazy mapcat
requires either fixing apply to handle lazy spreading (Step 4)
or rewriting defrecord to avoid mapcat entirely.
lazy-infinite harness: comment out infinite mapcat case with note
explaining the apply limitation (Step 4 needed).
Gates verified:
conformance 229/229 x3 (interpret/compile/self-host)
lazy-infinite 18/18
specs 32/32 files, 0 failures
xml-seq added to 20-coll.clj, matching Clojure reference:
(tree-seq (complement string?) (comp seq :content) root)
tree-seq kept eager with lazy version documented in comments.
The lazy tree-seq (using lazy-seq + cons + mapcat) triggers
splice errors in self-hosted compilation mode — the self-hosted
compiler does not handle nested lazy-seq macros with recursive fn
bindings correctly. The mapcat overlay in 10-seq.clj has the same
limitation. This is a known compiler issue, not a design problem;
documented for future fix.
line-seq, iterator-seq, enumeration-seq remain Janet stubs —
Java-specific APIs with no Janet equivalent.
flatten already correct in Clojure overlay (unchanged).
mapcat in 10-seq.clj now uses a defn- mapcat-step helper that walks
lazy map results element-by-element (cons + lazy-seq), with explicit
arities for 1/2/3/4+ colls. The 4+-coll arity still uses apply to
spread colls to map, but apply no longer forces the inner map result.
Previously (apply concat (apply map f colls)) forced the lazy map
result through core-apply realize-for-iteration, which hung on
infinite inputs. The new step-based impl walks one element at a time
via first/rest/seq primitives.
Re-enabled deferred mapcat infinite-input harness case. 19/19 pass.
Gates: lazy-infinite 19/19, conformance 229/229 interpret+self-host
(compile 228/229 — 1 pre-existing protocol error), specs 32/32.
ns-name is pure over get + symbol. aget/alength read jolt's mutable backing
arrays (nth/count), and aset writes a slot through jolt.host/ref-put!; both
handle the multi-dimensional form by walking. The array constructors
(object-array/make-array/to-array/...) stay native — they build the mutable
backing. Added array spec cases; ns-name already covered.
Pure reads of the future's :cached/:cancelled slots over future? + get.
future? stays native (deref/future-cancel/realized? call it), as do future-call
and future-cancel (OS threads). Covered by the existing futures spec.
vreset! sets the volatile's :val slot through the jolt.host/ref-put! primitive
added in the last batch; vswap! is pure over vreset! + get. The volatile!
constructor stays native (it builds the mutable box). Covered by the existing
state spec.
First host-primitive batch. Adds jolt.host/ref-put! — the minimal mutation
kernel (set/remove a key on a mutable reference cell) the overlay can't express
over core fns — and moves seven atom ops out of the seed:
swap-vals!/reset-vals!/compare-and-set! compose the native swap!/reset!/deref
(which already validate and notify watches); get-validator is a slot read;
add-watch/remove-watch/set-validator! mutate the atom (or its watches table)
via ref-put!.
atom/swap!/reset!/deref and atom-validate/atom-notify-watches stay native — the
compiler depends on them and they're hot. compare-and-set! keeps value-equality
semantics, matching the prior Janet behavior. Covered by the existing state spec.
atom?/volatile?/reader-conditional?/tagged-literal?/record?/chunked-seq? all
just read a value's kind from :jolt/type (records from :jolt/deftype), and get
returns nil on non-tables, so the predicates are pure over get with no host
surface. Constructors stay native. delay?/transient? keep their Janet defns —
they have internal callers (force, conj/count/persistent! etc.). chunked-seq?
is always false (no chunked seqs until Phase 5). Added a tagged-value spec
group covering positive and negative cases for each.
The ex-info value exposes :jolt/type/:message/:data/:cause via get (it's the
caught value too), so the accessors are pure over get — no host surface. The
constructor (ex-info) stays in Janet since it wires into throw. A thrown
non-ex-info arrives wrapped as {:jolt/type :jolt/exception :value v}; the
overlay unwraps that, matching the old Janet helper. ex-cause now unwraps and
returns nil (not false) on non-ex values, matching Clojure. Added non-ex-info
and string regression cases.
reduce-kv is pure over reduce/keys/get/nth, so it moves to 20-coll with no
host surface. Both branches fold through reduce, which fixes two latent bugs
in the Janet version: on a vector it saw the pvec as a table and folded over
its internal keys instead of the indices, and it ignored reduced entirely.
nil folds to init. Added vector-index, reduced, and nil spec cases plus a
3-mode conformance case for the vector fix.
First Phase 4 batch: the host-coupled fns whose logic is pure composition of
existing core primitives, so no new jolt.host surface is needed. vary-meta is
just (with-meta obj (apply f (meta obj) args)); namespace-munge is a char map
over str. Both land in the 20-coll tier; the Janet core-* defns and bindings
are gone. Added namespace-munge spec cases and a vary-meta extra-args case.
The heavily host-coupled fns (atom/var/transient internals, arrays, proxy,
futures) stay in Janet pending a thin host-primitive expansion.
Last core macro still in Janet. Goes in 00-syntax (not 30-macros with its
if-let/if-some/when-some siblings) because 20-coll's not-empty uses it and
20-coll loads before 30. Via `let` so the binding form may destructure,
matching Clojure — the old Janet version used let* and wouldn't.
Drops core-when-let + the sym* helper + the intern entry, and empties
core-macro-names (no seed binding is a macro anymore).
Port clojure.core/destructure into jolt-core/clojure/core/00-syntax.clj
and drop core-destructure (plus the d-* helpers) from core.janet. The
expander is now self-hosted, shrinking the seed per the jolt-1j0 epic.
It's def+fn* rather than defn because defn isn't defined yet at that
point in the syntax tier. Only the let macro consumes its output, so let
now splices [~@(destructure bindings)] to keep a tuple binding form.
Also fix a gap the old Janet version papered over via Janet's
keyword->string: :keys accepts keyword elements ({:keys [:major :minor]}),
so use name/namespace for the local + ns instead of str (which keeps the
colon). This was breaking sci's impl/namespaces.cljc. Added spec cases.
fn now desugars destructuring params like Clojure's maybe-destructured: each
non-symbol param becomes a gensym and the body is wrapped in a let that rebinds the
pattern, so fn* only ever sees plain params and the COMPILER handles it (it rejected
patterns before, falling back to the interpreter). loop follows Clojure too: gensym
one loop var per binding, loop* over those, destructure via an inner let, with an
outer let so later inits see earlier destructured names — recur arity stays correct.
Two representation gotchas: build the param/binding vectors via [~@..] so they're
tuple forms (conj yields a pvec the analyzer rejects), and use (symbol (str
(gensym))) since a bare (gensym) in an overlay macro body is a Janet symbol the
destructurer rejects.
Closes the fn/loop destructuring gaps. conformance 228x3, fixpoint, clojure-test-
suite 3930, full suite green, bench flat.
lazy-seq wraps its body in (make-lazy-seq (fn* [] (coll->cells (do ...)))); lazy-cat
wraps each coll in lazy-seq and concats (concat is already lazy). Both user-facing,
in 30-macros. Also drop two now-stale comments. Laziness preserved (lazy-seqs-spec
20/20, including infinite/self-referential).
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
defprotocol/defrecord/extend-type/extend-protocol/reify + the extend/proxy/
definterface stubs move to 30-macros (user-facing, not used by the compiler). They
emit Jolt's protocol/type special forms (protocol-dispatch/register-method/
make-reified/deftype).
The one subtlety: a protocol value is a :jolt/type-tagged struct, and the
interpreter can't tell an embedded opaque value from a tagged map literal being
constructed. So defprotocol builds it via a make-protocol fn call (exposed from
core) instead of an embedded literal — a fn result evaluates normally and even
compiles. reify emits a {kw (fn* ...)} map literal that make-reified evaluates
(build-eval-map yields a struct it can iterate, unlike a hash-map phm). defrecord
vecs its spliced field-let bindings (a lazy mapcat seq won't splice) and uses an
explicit fresh-sym for the map-> param (auto-gensym doesn't cross nested
syntax-quotes).
protocols-spec 21/21, conformance 228x3, fixpoint, clojure-test-suite 3930, full
suite green.
defn drops an optional leading docstring/attr-map then emits (def name (fn* ...)).
Single- and multi-arity both reduce to (fn* ~@body) so no arity branching is needed.
map? is true for symbol forms in Jolt, so the attr-map strip is guarded with symbol?.
defn- delegates to defn (privacy isn't enforced, as in Clojure's own defn-). Placed
before fresh-sym, which is itself a defn- now.
All five fundamental macros (fn/let/loop/defn/defn-) are now in the overlay.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
let -> let* with destructuring pre-expanded via destructure (now exposed as a
clojure.core fn, which it is in Clojure too) so the compiler sees plain bindings —
analyze-bindings rejects patterns as uncompilable. loop -> loop* with raw bindings,
matching the prior Janet macro: loop can't pre-destructure without breaking recur
arity, so the interpreter handles pattern loops and the compiler falls back.
conformance 228x3, fixpoint, clojure-test-suite 3930.
fn -> fn* is a one-line head-swap (the analyzer treats fn* as the primitive). It's
in 00-syntax since the analyzer, kernel, and other overlay macro bodies all use fn.
First of the fundamental macros to move.
conformance 228x3, fixpoint, clojure-test-suite 3930.
Port of core-for: desugar a comprehension to nested map/mapcat over the binding
colls, :let -> let*, :while -> take-while on the coll. Lives in 00-syntax because
doseq (already there) expands to it; the macro body uses only kernel/seed fns so it
runs at analyzer-build time. doseq no longer depends on a Janet macro.
Fixed a latent bug the Janet macro had: :when wrapped the inner form in (list ...)
unconditionally, but for an outer binding group the inner form is already a seq, so
mapcat produced a seq-of-seqs instead of flattening. e.g.
(for [x [0 1] :when (odd? x) y [:a :b]] [x y])
gave ((... ...)) instead of ([1 :a] [1 :b]). Now the (list ...) wrap is only applied
to the last group's scalar body; outer groups contribute their seq directly. Added
:let+:when, multi-group :when, and destructuring regression cases.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green, bench flat.
Same shortcut as the prior Janet macro — realize a for comprehension with count
for side effects, return nil — so for keeps handling :when/:let/:while and multiple
bindings. Lives in 00-syntax because the analyzer uses doseq (analyze-try). Added
:when/:while/:let/returns-nil regression cases.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
The real bug behind the case/cond-> fixpoint regression wasn't gensym — it was
build ordering. A top-level defn in a pre-kernel tier (the fresh-sym helper these
macros need) gets compiled in compile mode, and that lazily builds the self-hosted
analyzer via ensure-analyzer. 00-syntax loads before 00-kernel, so the analyzer
was built against a core missing mapv/second/peek/...: its references to them were
interned as forward-ref nil cells in jolt.analyzer. Those nil cells then shadowed
the real clojure.core defs when the analyzer rebuilt itself (stage2), so the
analyzer's own mapv calls went to nil — 'Cannot call nil as a function'. Only
variadic-heavy paths (juxt+mapv) surfaced it.
build-compiler! already documented that the kernel must be loaded first; nothing
enforced it. Gate ensure-analyzer on a :kernel-ready? flag set after the kernel
tier loads. A pre-kernel compile now falls back to the interpreter (compile-and-eval
already handles that) instead of building the analyzer too early. fresh-sym ends up
interpreted during 00-syntax load, which is fine.
With ordering fixed, case and cond-> move to 00-syntax (they need a real gensym, so
syntax-quote auto-gensym alone won't do). conformance 228x3, fixpoint stage1==2==3,
clojure-test-suite 3930, full suite green, bench flat.
Threading macros (recursive; the expand-once cache makes that free) and declare
(a no-op on Jolt — forward refs resolve via pending cells). They live in 00-syntax
because the analyzer itself uses -> and declare; validated by conformance 228x3
(the bootstrap-compiled analyzer expands them).
conformance 228/228 x3, clojure-test-suite 3930.
The hot control macros now live in the Clojure syntax tier. This was blocked
before: as interpreted overlay macros they re-expanded on every eval, timing out
a battery file (3930->3911). With the expand-once macro cache (prior commit) they
expand a single time with zero runtime cost, so moving them is free.
conformance 228/228 x3, clojure-test-suite 3930 (9 timeouts, not 10), full suite
green, bench flat (overlay vs janet macros within noise).
Following the canonical read->macroexpand->compile model (Clojure LispReader /
tools.reader): a syntax-quote is lowered to plain construction code instead of
being interpreted. Adds form builders (__sqcat -> array, __sqvec -> tuple,
__sqmap, __sq1) and syntax-quote-lower (mirrors the interpreter's syntax-quote*/
sq-symbol exactly: foo# auto-gensym, core/special unqualified, else qualify;
~ -> expr, ~@ -> splice). The analyzer now handles syntax-quote as a special:
lower then analyze the result. The interpreter keeps the self-contained
syntax-quote* (no core dependency, works in a bare ctx); the two are cross-checked
by conformance interpret-vs-compile.
Effect: a backtick body compiles (1.96s -> 1.19s/200k) instead of falling back to
the interpreter. Next: compile defmacro expanders for the full macro speedup.
conformance 228/228 x3, full suite green.
New 00-syntax tier loaded FIRST (before 00-kernel), interpreted defmacros, so the
control macros the compiler and every later tier depend on can live in Clojure.
Validated by moving when: the kernel tier, self-hosted analyzer and seq/coll
tiers all compile against the overlay when. Constraint: syntax-tier macros may use
only special forms + core-renames seed primitives (not second/peek/kernel fns).
conformance 228/228 x3, full suite green.
Phase 3 batch 6 (jolt-1j0). binding installs an array-map var->value thread
frame and restores it on exit via try/finally. Dynamic rebinding is seen by
called fns. This completes the cleanly-portable safe macros (19 total).
conformance 228/228 x3, full suite green.
Phase 3 batch 5 (jolt-1j0). condp with a recursive emit building the nested if
chain, including the test :>> result-fn form and the trailing default. This
exhausts the cleanly-portable safe macros.
conformance 228/228 x3, full suite green.
Phase 3 batch 4 (jolt-1j0), 5 macros. cond->> (thread-last) is safe; cond->
stays (compiler uses it). delay/future expand to make-delay/future-call host
fns; letfn builds its fn* binding via a template (cons/list in a macro body make
a plist the evaluator can't call as a form).
conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
Phase 3 batch 3 (jolt-1j0), 5 macros. Establishes the macro-body toolkit:
fresh-sym = (symbol (str (gensym))) for a shared explicit jolt symbol (a bare
(gensym) returns a Janet symbol the destructurer rejects), and splicing binding
pairs into a TEMPLATE vector so core-let sees a tuple form, not a runtime pvec.
when-first stays first-based (lazy-safe): Clojure's (seq coll) form realizes an
infinite coll like (repeat nil) under Jolt's eager evaluator and hangs — the
per-batch battery caught it (when_first.cljc). Clojure-correct seq form waits on
Phase 5 laziness.
conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
Phase 3 batch 2 (jolt-1j0), 5 user-facing macros as defmacro + syntax-quote.
The conditional-binding macros use the auto-gensym temp# idiom so the name binds
only in the taken branch (the else/empty branch sees the outer scope, carrying
forward the earlier scope fix). when-let stays in Janet for now — 20-coll uses
it, so it must remain available before the macro tier loads.
Note: gensym in a macro body resolves to Janet's 0-arity builtin, so explicit
(gensym "prefix") fails — use template auto-gensym (foo#) instead.
conformance 228/228 x3, full suite green (incl. the if-let scope regression spec,
now exercising the overlay macro), clojure-test-suite 3930, bench A/B flat.
New 30-macros tier (registered after 20-coll) for user-facing macros expressed as
defmacro + syntax-quote instead of hand-built Janet form-transformers. Validated
end to end: overlay-defined macros expand in interpret, compile AND self-host
modes (conformance 228/228 x3).
Moved comment and if-not; removed their Janet core-X fns + core-macro-names
entries. Note: Jolt defmacro is single-arity, so multi-arglist macros become one
arglist with & rest / destructuring (if-not uses [test then & [else]]). Macros
used by the compiler or earlier tiers (and/or/when/cond/case/cond->/->/declare/
doseq/when-let) stay in Janet until a load-order story exists for them.
full suite green, clojure-test-suite 3930.
Eighth pure-fn batch (jolt-1j0 phase 2), 2 leaf fns. dedupe is eager consecutive
dedup (no transducer arity, as before); seq-to-map-for-destructuring is the
internal &{:keys} helper. Largely exhausts the easy pure-eager tier.
conformance 228/228 x3, full suite green (incl. destructuring).
