The lazy-seq macro (30-macros.clj) is not available when 20-coll
loads. Use explicit (make-lazy-seq (fn* [] (coll->cells ...)))
expansion. Also fix outer cons leaking raw @[val thunk] cell.
22/22 lazy-infinite, conformance 229 interpret + self-host.
The git stash pop restored the old lazy-seq version of dedupe.
Re-applied the make-lazy-seq + coll->cells fix for the 20-coll
tier where lazy-seq macro is not yet available.
core-trampoline calls apply internally. In compile mode, apply must
be in core-renames for the compiler to emit a direct call to
core-apply. Without this, apply resolves to nil at compile time.
Add "trampoline" "core-trampoline" to core-renames so compile
mode emits direct calls. Without this entry the compiler falls
back to resolving trampoline in Clojure namespace, which fails
since trampoline was removed from overlay.
Fix 20-coll.clj extra close paren from trampoline removal.
Overlay trampoline fails arity dispatch — (trampoline f 5) with [f]
and [f & args] arities dispatches incorrectly in Jolt. The Janet
core-trampoline is 4 lines and works correctly. Kept native.
Results: 22/22 lazy-infinite, 229x3 conformance, 32/32 specs.
dedupe: use make-lazy-seq + coll->cells directly — lazy-seq
macro is not available at 20-coll tier (defined in 30-macros.clj).
trampoline: use ifn? instead of fn? (not available), fix
single-arity to call (f) directly instead of recursive (trampoline f (f)).
rand-int: removed from overlay, stays native in core.janet
(math/floor and math/random not available in Clojure namespace).
Fix & rest destructuring (evaluator.janet): when the original
value is a lazy-seq, derive & rest by walking ls-rest vi steps
instead of slicing the eagerly-realized array from d-realize.
Rewrite core-mapcat as a lazy state machine (core.janet): step
through input colls one element at a time, call f on each tuple,
then yield from f's result collection. No apply-forcing —
all input colls are walked lazily via lazy-from + seq-done?.
Add mapcat to core-renames (compiler.janet) and remove from
Clojure overlay (10-seq.clj) — compile mode now emits direct
calls to the lazy core-mapcat, fixing the pre-existing
protocol-on-record compile error.
Tests: re-enabled mapcat infinite harness case, added 2 & rest
laziness cases. 21/21 lazy-infinite, 229x3 conformance, 32/32 specs.
core-drop-while (core.janet): return cell via realize-ls instead
of raw LazySeq. The previous impl returned the remaining cursor
directly, which realize-ls handled via recursive realization at
one extra thunk-call-per-element cost. Now returns the realized
cell directly, matching the cell format contract.
mapcat (10-seq.clj): revert to standard (apply concat (apply map
f colls)). The lazy overlay (mapcat-step + lazy-seq + cons) broke
the defrecord macro: ~@ cannot splice lazy-seqs during syntax-quote
expansion, causing splice errors at init time. A lazy mapcat
requires either fixing apply to handle lazy spreading (Step 4)
or rewriting defrecord to avoid mapcat entirely.
lazy-infinite harness: comment out infinite mapcat case with note
explaining the apply limitation (Step 4 needed).
Gates verified:
conformance 229/229 x3 (interpret/compile/self-host)
lazy-infinite 18/18
specs 32/32 files, 0 failures
xml-seq added to 20-coll.clj, matching Clojure reference:
(tree-seq (complement string?) (comp seq :content) root)
tree-seq kept eager with lazy version documented in comments.
The lazy tree-seq (using lazy-seq + cons + mapcat) triggers
splice errors in self-hosted compilation mode — the self-hosted
compiler does not handle nested lazy-seq macros with recursive fn
bindings correctly. The mapcat overlay in 10-seq.clj has the same
limitation. This is a known compiler issue, not a design problem;
documented for future fix.
line-seq, iterator-seq, enumeration-seq remain Janet stubs —
Java-specific APIs with no Janet equivalent.
flatten already correct in Clojure overlay (unchanged).
mapcat in 10-seq.clj now uses a defn- mapcat-step helper that walks
lazy map results element-by-element (cons + lazy-seq), with explicit
arities for 1/2/3/4+ colls. The 4+-coll arity still uses apply to
spread colls to map, but apply no longer forces the inner map result.
Previously (apply concat (apply map f colls)) forced the lazy map
result through core-apply realize-for-iteration, which hung on
infinite inputs. The new step-based impl walks one element at a time
via first/rest/seq primitives.
Re-enabled deferred mapcat infinite-input harness case. 19/19 pass.
Gates: lazy-infinite 19/19, conformance 229/229 interpret+self-host
(compile 228/229 — 1 pre-existing protocol error), specs 32/32.
Step-by-step plan for jolt-c09 (Phase 5 of the clojure.core migration):
current state of the LazySeq machinery and the eager gaps, the cardinal
laziness rules, leaf-first transformer conversion order, realization-boundary
audit, the representation decision (lazy-seq vs eager-vector for map over a
vector), and a testing strategy built around a deadlined subprocess harness
(infinite realizations are CPU-bound and uninterruptible in-process).
ns-name is pure over get + symbol. aget/alength read jolt's mutable backing
arrays (nth/count), and aset writes a slot through jolt.host/ref-put!; both
handle the multi-dimensional form by walking. The array constructors
(object-array/make-array/to-array/...) stay native — they build the mutable
backing. Added array spec cases; ns-name already covered.
Pure reads of the future's :cached/:cancelled slots over future? + get.
future? stays native (deref/future-cancel/realized? call it), as do future-call
and future-cancel (OS threads). Covered by the existing futures spec.
vreset! sets the volatile's :val slot through the jolt.host/ref-put! primitive
added in the last batch; vswap! is pure over vreset! + get. The volatile!
constructor stays native (it builds the mutable box). Covered by the existing
state spec.
First host-primitive batch. Adds jolt.host/ref-put! — the minimal mutation
kernel (set/remove a key on a mutable reference cell) the overlay can't express
over core fns — and moves seven atom ops out of the seed:
swap-vals!/reset-vals!/compare-and-set! compose the native swap!/reset!/deref
(which already validate and notify watches); get-validator is a slot read;
add-watch/remove-watch/set-validator! mutate the atom (or its watches table)
via ref-put!.
atom/swap!/reset!/deref and atom-validate/atom-notify-watches stay native — the
compiler depends on them and they're hot. compare-and-set! keeps value-equality
semantics, matching the prior Janet behavior. Covered by the existing state spec.
atom?/volatile?/reader-conditional?/tagged-literal?/record?/chunked-seq? all
just read a value's kind from :jolt/type (records from :jolt/deftype), and get
returns nil on non-tables, so the predicates are pure over get with no host
surface. Constructors stay native. delay?/transient? keep their Janet defns —
they have internal callers (force, conj/count/persistent! etc.). chunked-seq?
is always false (no chunked seqs until Phase 5). Added a tagged-value spec
group covering positive and negative cases for each.
The ex-info value exposes :jolt/type/:message/:data/:cause via get (it's the
caught value too), so the accessors are pure over get — no host surface. The
constructor (ex-info) stays in Janet since it wires into throw. A thrown
non-ex-info arrives wrapped as {:jolt/type :jolt/exception :value v}; the
overlay unwraps that, matching the old Janet helper. ex-cause now unwraps and
returns nil (not false) on non-ex values, matching Clojure. Added non-ex-info
and string regression cases.
reduce-kv is pure over reduce/keys/get/nth, so it moves to 20-coll with no
host surface. Both branches fold through reduce, which fixes two latent bugs
in the Janet version: on a vector it saw the pvec as a table and folded over
its internal keys instead of the indices, and it ignored reduced entirely.
nil folds to init. Added vector-index, reduced, and nil spec cases plus a
3-mode conformance case for the vector fix.
First Phase 4 batch: the host-coupled fns whose logic is pure composition of
existing core primitives, so no new jolt.host surface is needed. vary-meta is
just (with-meta obj (apply f (meta obj) args)); namespace-munge is a char map
over str. Both land in the 20-coll tier; the Janet core-* defns and bindings
are gone. Added namespace-munge spec cases and a vary-meta extra-args case.
The heavily host-coupled fns (atom/var/transient internals, arrays, proxy,
futures) stay in Janet pending a thin host-primitive expansion.
Last core macro still in Janet. Goes in 00-syntax (not 30-macros with its
if-let/if-some/when-some siblings) because 20-coll's not-empty uses it and
20-coll loads before 30. Via `let` so the binding form may destructure,
matching Clojure — the old Janet version used let* and wouldn't.
Drops core-when-let + the sym* helper + the intern entry, and empties
core-macro-names (no seed binding is a macro anymore).
Port clojure.core/destructure into jolt-core/clojure/core/00-syntax.clj
and drop core-destructure (plus the d-* helpers) from core.janet. The
expander is now self-hosted, shrinking the seed per the jolt-1j0 epic.
It's def+fn* rather than defn because defn isn't defined yet at that
point in the syntax tier. Only the let macro consumes its output, so let
now splices [~@(destructure bindings)] to keep a tuple binding form.
Also fix a gap the old Janet version papered over via Janet's
keyword->string: :keys accepts keyword elements ({:keys [:major :minor]}),
so use name/namespace for the local + ns instead of str (which keeps the
colon). This was breaking sci's impl/namespaces.cljc. Added spec cases.
fn now desugars destructuring params like Clojure's maybe-destructured: each
non-symbol param becomes a gensym and the body is wrapped in a let that rebinds the
pattern, so fn* only ever sees plain params and the COMPILER handles it (it rejected
patterns before, falling back to the interpreter). loop follows Clojure too: gensym
one loop var per binding, loop* over those, destructure via an inner let, with an
outer let so later inits see earlier destructured names — recur arity stays correct.
Two representation gotchas: build the param/binding vectors via [~@..] so they're
tuple forms (conj yields a pvec the analyzer rejects), and use (symbol (str
(gensym))) since a bare (gensym) in an overlay macro body is a Janet symbol the
destructurer rejects.
Closes the fn/loop destructuring gaps. conformance 228x3, fixpoint, clojure-test-
suite 3930, full suite green, bench flat.
lazy-seq wraps its body in (make-lazy-seq (fn* [] (coll->cells (do ...)))); lazy-cat
wraps each coll in lazy-seq and concats (concat is already lazy). Both user-facing,
in 30-macros. Also drop two now-stale comments. Laziness preserved (lazy-seqs-spec
20/20, including infinite/self-referential).
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
defprotocol/defrecord/extend-type/extend-protocol/reify + the extend/proxy/
definterface stubs move to 30-macros (user-facing, not used by the compiler). They
emit Jolt's protocol/type special forms (protocol-dispatch/register-method/
make-reified/deftype).
The one subtlety: a protocol value is a :jolt/type-tagged struct, and the
interpreter can't tell an embedded opaque value from a tagged map literal being
constructed. So defprotocol builds it via a make-protocol fn call (exposed from
core) instead of an embedded literal — a fn result evaluates normally and even
compiles. reify emits a {kw (fn* ...)} map literal that make-reified evaluates
(build-eval-map yields a struct it can iterate, unlike a hash-map phm). defrecord
vecs its spliced field-let bindings (a lazy mapcat seq won't splice) and uses an
explicit fresh-sym for the map-> param (auto-gensym doesn't cross nested
syntax-quotes).
protocols-spec 21/21, conformance 228x3, fixpoint, clojure-test-suite 3930, full
suite green.
defn drops an optional leading docstring/attr-map then emits (def name (fn* ...)).
Single- and multi-arity both reduce to (fn* ~@body) so no arity branching is needed.
map? is true for symbol forms in Jolt, so the attr-map strip is guarded with symbol?.
defn- delegates to defn (privacy isn't enforced, as in Clojure's own defn-). Placed
before fresh-sym, which is itself a defn- now.
All five fundamental macros (fn/let/loop/defn/defn-) are now in the overlay.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
let -> let* with destructuring pre-expanded via destructure (now exposed as a
clojure.core fn, which it is in Clojure too) so the compiler sees plain bindings —
analyze-bindings rejects patterns as uncompilable. loop -> loop* with raw bindings,
matching the prior Janet macro: loop can't pre-destructure without breaking recur
arity, so the interpreter handles pattern loops and the compiler falls back.
conformance 228x3, fixpoint, clojure-test-suite 3930.
fn -> fn* is a one-line head-swap (the analyzer treats fn* as the primitive). It's
in 00-syntax since the analyzer, kernel, and other overlay macro bodies all use fn.
First of the fundamental macros to move.
conformance 228x3, fixpoint, clojure-test-suite 3930.
Port of core-for: desugar a comprehension to nested map/mapcat over the binding
colls, :let -> let*, :while -> take-while on the coll. Lives in 00-syntax because
doseq (already there) expands to it; the macro body uses only kernel/seed fns so it
runs at analyzer-build time. doseq no longer depends on a Janet macro.
Fixed a latent bug the Janet macro had: :when wrapped the inner form in (list ...)
unconditionally, but for an outer binding group the inner form is already a seq, so
mapcat produced a seq-of-seqs instead of flattening. e.g.
(for [x [0 1] :when (odd? x) y [:a :b]] [x y])
gave ((... ...)) instead of ([1 :a] [1 :b]). Now the (list ...) wrap is only applied
to the last group's scalar body; outer groups contribute their seq directly. Added
:let+:when, multi-group :when, and destructuring regression cases.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green, bench flat.
Same shortcut as the prior Janet macro — realize a for comprehension with count
for side effects, return nil — so for keeps handling :when/:let/:while and multiple
bindings. Lives in 00-syntax because the analyzer uses doseq (analyze-try). Added
:when/:while/:let/returns-nil regression cases.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
The real bug behind the case/cond-> fixpoint regression wasn't gensym — it was
build ordering. A top-level defn in a pre-kernel tier (the fresh-sym helper these
macros need) gets compiled in compile mode, and that lazily builds the self-hosted
analyzer via ensure-analyzer. 00-syntax loads before 00-kernel, so the analyzer
was built against a core missing mapv/second/peek/...: its references to them were
interned as forward-ref nil cells in jolt.analyzer. Those nil cells then shadowed
the real clojure.core defs when the analyzer rebuilt itself (stage2), so the
analyzer's own mapv calls went to nil — 'Cannot call nil as a function'. Only
variadic-heavy paths (juxt+mapv) surfaced it.
build-compiler! already documented that the kernel must be loaded first; nothing
enforced it. Gate ensure-analyzer on a :kernel-ready? flag set after the kernel
tier loads. A pre-kernel compile now falls back to the interpreter (compile-and-eval
already handles that) instead of building the analyzer too early. fresh-sym ends up
interpreted during 00-syntax load, which is fine.
With ordering fixed, case and cond-> move to 00-syntax (they need a real gensym, so
syntax-quote auto-gensym alone won't do). conformance 228x3, fixpoint stage1==2==3,
clojure-test-suite 3930, full suite green, bench flat.
Threading macros (recursive; the expand-once cache makes that free) and declare
(a no-op on Jolt — forward refs resolve via pending cells). They live in 00-syntax
because the analyzer itself uses -> and declare; validated by conformance 228x3
(the bootstrap-compiled analyzer expands them).
conformance 228/228 x3, clojure-test-suite 3930.
The hot control macros now live in the Clojure syntax tier. This was blocked
before: as interpreted overlay macros they re-expanded on every eval, timing out
a battery file (3930->3911). With the expand-once macro cache (prior commit) they
expand a single time with zero runtime cost, so moving them is free.
conformance 228/228 x3, clojure-test-suite 3930 (9 timeouts, not 10), full suite
green, bench flat (overlay vs janet macros within noise).
The interpreter re-expanded a macro call on every evaluation — so a macro in a
hot loop re-expanded each iteration (the runtime cost, and why moving the hot
macros to the overlay timed out battery files). Now each macro CALL form expands
once, keyed by form identity, and the macro-free expansion is reused; this is the
proper Lisp model (macroexpansion is a compile-time step). Also gives compile-once
gensym semantics — a foo# auto-gensym is now fixed across calls instead of fresh
per re-expansion. Cache cleared when a macro is (re)defined.
conformance 228/228 x3, full suite green.
A defmacro now compiles its expander to a native Janet fn — built as
(fn* args body...) and run through the self-hosted pipeline (its backtick body is
now compilable, prior commit) — instead of an interpreted closure. Macro
expansion happens at compile time with zero runtime cost, the proper Lisp model.
Wiring: evaluator exposes a macro-compile-hook the api sets to backend/
try-compile-fn; the defmacro handler uses the compiled expander when available,
falling back to the interpreted closure when the analyzer isn't built yet (early
tiers) or the body uses &env/&form (no such params on the compiled fn).
conformance 228/228 x3 (incl. REPL defmacro + gensym hygiene), full suite green.
Following the canonical read->macroexpand->compile model (Clojure LispReader /
tools.reader): a syntax-quote is lowered to plain construction code instead of
being interpreted. Adds form builders (__sqcat -> array, __sqvec -> tuple,
__sqmap, __sq1) and syntax-quote-lower (mirrors the interpreter's syntax-quote*/
sq-symbol exactly: foo# auto-gensym, core/special unqualified, else qualify;
~ -> expr, ~@ -> splice). The analyzer now handles syntax-quote as a special:
lower then analyze the result. The interpreter keeps the self-contained
syntax-quote* (no core dependency, works in a bare ctx); the two are cross-checked
by conformance interpret-vs-compile.
Effect: a backtick body compiles (1.96s -> 1.19s/200k) instead of falling back to
the interpreter. Next: compile defmacro expanders for the full macro speedup.
conformance 228/228 x3, full suite green.
New 00-syntax tier loaded FIRST (before 00-kernel), interpreted defmacros, so the
control macros the compiler and every later tier depend on can live in Clojure.
Validated by moving when: the kernel tier, self-hosted analyzer and seq/coll
tiers all compile against the overlay when. Constraint: syntax-tier macros may use
only special forms + core-renames seed primitives (not second/peek/kernel fns).
conformance 228/228 x3, full suite green.
Phase 3 batch 6 (jolt-1j0). binding installs an array-map var->value thread
frame and restores it on exit via try/finally. Dynamic rebinding is seen by
called fns. This completes the cleanly-portable safe macros (19 total).
conformance 228/228 x3, full suite green.
Phase 3 batch 5 (jolt-1j0). condp with a recursive emit building the nested if
chain, including the test :>> result-fn form and the trailing default. This
exhausts the cleanly-portable safe macros.
conformance 228/228 x3, full suite green.
Phase 3 batch 4 (jolt-1j0), 5 macros. cond->> (thread-last) is safe; cond->
stays (compiler uses it). delay/future expand to make-delay/future-call host
fns; letfn builds its fn* binding via a template (cons/list in a macro body make
a plist the evaluator can't call as a form).
conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
Phase 3 batch 3 (jolt-1j0), 5 macros. Establishes the macro-body toolkit:
fresh-sym = (symbol (str (gensym))) for a shared explicit jolt symbol (a bare
(gensym) returns a Janet symbol the destructurer rejects), and splicing binding
pairs into a TEMPLATE vector so core-let sees a tuple form, not a runtime pvec.
when-first stays first-based (lazy-safe): Clojure's (seq coll) form realizes an
infinite coll like (repeat nil) under Jolt's eager evaluator and hangs — the
per-batch battery caught it (when_first.cljc). Clojure-correct seq form waits on
Phase 5 laziness.
conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
Phase 3 batch 2 (jolt-1j0), 5 user-facing macros as defmacro + syntax-quote.
The conditional-binding macros use the auto-gensym temp# idiom so the name binds
only in the taken branch (the else/empty branch sees the outer scope, carrying
forward the earlier scope fix). when-let stays in Janet for now — 20-coll uses
it, so it must remain available before the macro tier loads.
Note: gensym in a macro body resolves to Janet's 0-arity builtin, so explicit
(gensym "prefix") fails — use template auto-gensym (foo#) instead.
conformance 228/228 x3, full suite green (incl. the if-let scope regression spec,
now exercising the overlay macro), clojure-test-suite 3930, bench A/B flat.
New 30-macros tier (registered after 20-coll) for user-facing macros expressed as
defmacro + syntax-quote instead of hand-built Janet form-transformers. Validated
end to end: overlay-defined macros expand in interpret, compile AND self-host
modes (conformance 228/228 x3).
Moved comment and if-not; removed their Janet core-X fns + core-macro-names
entries. Note: Jolt defmacro is single-arity, so multi-arglist macros become one
arglist with & rest / destructuring (if-not uses [test then & [else]]). Macros
used by the compiler or earlier tiers (and/or/when/cond/case/cond->/->/declare/
doseq/when-let) stay in Janet until a load-order story exists for them.
full suite green, clojure-test-suite 3930.
Eighth pure-fn batch (jolt-1j0 phase 2), 2 leaf fns. dedupe is eager consecutive
dedup (no transducer arity, as before); seq-to-map-for-destructuring is the
internal &{:keys} helper. Largely exhausts the easy pure-eager tier.
conformance 228/228 x3, full suite green (incl. destructuring).
Sixth pure-fn batch (jolt-1j0 phase 2), 3 leaf fns, now with regression tests
per the per-batch workflow. reductions is the canonical form, so (reductions f [])
calls (f) -> [0] instead of the prior []; tree-seq is eager pre-order DFS.
conformance 228/228 x3, clojure-test-suite 3929, full suite green, bench A/B flat.
