lazy-seq wraps its body in (make-lazy-seq (fn* [] (coll->cells (do ...)))); lazy-cat
wraps each coll in lazy-seq and concats (concat is already lazy). Both user-facing,
in 30-macros. Also drop two now-stale comments. Laziness preserved (lazy-seqs-spec
20/20, including infinite/self-referential).
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
defprotocol/defrecord/extend-type/extend-protocol/reify + the extend/proxy/
definterface stubs move to 30-macros (user-facing, not used by the compiler). They
emit Jolt's protocol/type special forms (protocol-dispatch/register-method/
make-reified/deftype).
The one subtlety: a protocol value is a :jolt/type-tagged struct, and the
interpreter can't tell an embedded opaque value from a tagged map literal being
constructed. So defprotocol builds it via a make-protocol fn call (exposed from
core) instead of an embedded literal — a fn result evaluates normally and even
compiles. reify emits a {kw (fn* ...)} map literal that make-reified evaluates
(build-eval-map yields a struct it can iterate, unlike a hash-map phm). defrecord
vecs its spliced field-let bindings (a lazy mapcat seq won't splice) and uses an
explicit fresh-sym for the map-> param (auto-gensym doesn't cross nested
syntax-quotes).
protocols-spec 21/21, conformance 228x3, fixpoint, clojure-test-suite 3930, full
suite green.
defn drops an optional leading docstring/attr-map then emits (def name (fn* ...)).
Single- and multi-arity both reduce to (fn* ~@body) so no arity branching is needed.
map? is true for symbol forms in Jolt, so the attr-map strip is guarded with symbol?.
defn- delegates to defn (privacy isn't enforced, as in Clojure's own defn-). Placed
before fresh-sym, which is itself a defn- now.
All five fundamental macros (fn/let/loop/defn/defn-) are now in the overlay.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
let -> let* with destructuring pre-expanded via destructure (now exposed as a
clojure.core fn, which it is in Clojure too) so the compiler sees plain bindings —
analyze-bindings rejects patterns as uncompilable. loop -> loop* with raw bindings,
matching the prior Janet macro: loop can't pre-destructure without breaking recur
arity, so the interpreter handles pattern loops and the compiler falls back.
conformance 228x3, fixpoint, clojure-test-suite 3930.
fn -> fn* is a one-line head-swap (the analyzer treats fn* as the primitive). It's
in 00-syntax since the analyzer, kernel, and other overlay macro bodies all use fn.
First of the fundamental macros to move.
conformance 228x3, fixpoint, clojure-test-suite 3930.
Port of core-for: desugar a comprehension to nested map/mapcat over the binding
colls, :let -> let*, :while -> take-while on the coll. Lives in 00-syntax because
doseq (already there) expands to it; the macro body uses only kernel/seed fns so it
runs at analyzer-build time. doseq no longer depends on a Janet macro.
Fixed a latent bug the Janet macro had: :when wrapped the inner form in (list ...)
unconditionally, but for an outer binding group the inner form is already a seq, so
mapcat produced a seq-of-seqs instead of flattening. e.g.
(for [x [0 1] :when (odd? x) y [:a :b]] [x y])
gave ((... ...)) instead of ([1 :a] [1 :b]). Now the (list ...) wrap is only applied
to the last group's scalar body; outer groups contribute their seq directly. Added
:let+:when, multi-group :when, and destructuring regression cases.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green, bench flat.
Same shortcut as the prior Janet macro — realize a for comprehension with count
for side effects, return nil — so for keeps handling :when/:let/:while and multiple
bindings. Lives in 00-syntax because the analyzer uses doseq (analyze-try). Added
:when/:while/:let/returns-nil regression cases.
conformance 228x3, fixpoint, clojure-test-suite 3930, full suite green.
The real bug behind the case/cond-> fixpoint regression wasn't gensym — it was
build ordering. A top-level defn in a pre-kernel tier (the fresh-sym helper these
macros need) gets compiled in compile mode, and that lazily builds the self-hosted
analyzer via ensure-analyzer. 00-syntax loads before 00-kernel, so the analyzer
was built against a core missing mapv/second/peek/...: its references to them were
interned as forward-ref nil cells in jolt.analyzer. Those nil cells then shadowed
the real clojure.core defs when the analyzer rebuilt itself (stage2), so the
analyzer's own mapv calls went to nil — 'Cannot call nil as a function'. Only
variadic-heavy paths (juxt+mapv) surfaced it.
build-compiler! already documented that the kernel must be loaded first; nothing
enforced it. Gate ensure-analyzer on a :kernel-ready? flag set after the kernel
tier loads. A pre-kernel compile now falls back to the interpreter (compile-and-eval
already handles that) instead of building the analyzer too early. fresh-sym ends up
interpreted during 00-syntax load, which is fine.
With ordering fixed, case and cond-> move to 00-syntax (they need a real gensym, so
syntax-quote auto-gensym alone won't do). conformance 228x3, fixpoint stage1==2==3,
clojure-test-suite 3930, full suite green, bench flat.
Threading macros (recursive; the expand-once cache makes that free) and declare
(a no-op on Jolt — forward refs resolve via pending cells). They live in 00-syntax
because the analyzer itself uses -> and declare; validated by conformance 228x3
(the bootstrap-compiled analyzer expands them).
conformance 228/228 x3, clojure-test-suite 3930.
The hot control macros now live in the Clojure syntax tier. This was blocked
before: as interpreted overlay macros they re-expanded on every eval, timing out
a battery file (3930->3911). With the expand-once macro cache (prior commit) they
expand a single time with zero runtime cost, so moving them is free.
conformance 228/228 x3, clojure-test-suite 3930 (9 timeouts, not 10), full suite
green, bench flat (overlay vs janet macros within noise).
Following the canonical read->macroexpand->compile model (Clojure LispReader /
tools.reader): a syntax-quote is lowered to plain construction code instead of
being interpreted. Adds form builders (__sqcat -> array, __sqvec -> tuple,
__sqmap, __sq1) and syntax-quote-lower (mirrors the interpreter's syntax-quote*/
sq-symbol exactly: foo# auto-gensym, core/special unqualified, else qualify;
~ -> expr, ~@ -> splice). The analyzer now handles syntax-quote as a special:
lower then analyze the result. The interpreter keeps the self-contained
syntax-quote* (no core dependency, works in a bare ctx); the two are cross-checked
by conformance interpret-vs-compile.
Effect: a backtick body compiles (1.96s -> 1.19s/200k) instead of falling back to
the interpreter. Next: compile defmacro expanders for the full macro speedup.
conformance 228/228 x3, full suite green.
New 00-syntax tier loaded FIRST (before 00-kernel), interpreted defmacros, so the
control macros the compiler and every later tier depend on can live in Clojure.
Validated by moving when: the kernel tier, self-hosted analyzer and seq/coll
tiers all compile against the overlay when. Constraint: syntax-tier macros may use
only special forms + core-renames seed primitives (not second/peek/kernel fns).
conformance 228/228 x3, full suite green.
Phase 3 batch 6 (jolt-1j0). binding installs an array-map var->value thread
frame and restores it on exit via try/finally. Dynamic rebinding is seen by
called fns. This completes the cleanly-portable safe macros (19 total).
conformance 228/228 x3, full suite green.
Phase 3 batch 5 (jolt-1j0). condp with a recursive emit building the nested if
chain, including the test :>> result-fn form and the trailing default. This
exhausts the cleanly-portable safe macros.
conformance 228/228 x3, full suite green.
Phase 3 batch 4 (jolt-1j0), 5 macros. cond->> (thread-last) is safe; cond->
stays (compiler uses it). delay/future expand to make-delay/future-call host
fns; letfn builds its fn* binding via a template (cons/list in a macro body make
a plist the evaluator can't call as a form).
conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
Phase 3 batch 3 (jolt-1j0), 5 macros. Establishes the macro-body toolkit:
fresh-sym = (symbol (str (gensym))) for a shared explicit jolt symbol (a bare
(gensym) returns a Janet symbol the destructurer rejects), and splicing binding
pairs into a TEMPLATE vector so core-let sees a tuple form, not a runtime pvec.
when-first stays first-based (lazy-safe): Clojure's (seq coll) form realizes an
infinite coll like (repeat nil) under Jolt's eager evaluator and hangs — the
per-batch battery caught it (when_first.cljc). Clojure-correct seq form waits on
Phase 5 laziness.
conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
Phase 3 batch 2 (jolt-1j0), 5 user-facing macros as defmacro + syntax-quote.
The conditional-binding macros use the auto-gensym temp# idiom so the name binds
only in the taken branch (the else/empty branch sees the outer scope, carrying
forward the earlier scope fix). when-let stays in Janet for now — 20-coll uses
it, so it must remain available before the macro tier loads.
Note: gensym in a macro body resolves to Janet's 0-arity builtin, so explicit
(gensym "prefix") fails — use template auto-gensym (foo#) instead.
conformance 228/228 x3, full suite green (incl. the if-let scope regression spec,
now exercising the overlay macro), clojure-test-suite 3930, bench A/B flat.
New 30-macros tier (registered after 20-coll) for user-facing macros expressed as
defmacro + syntax-quote instead of hand-built Janet form-transformers. Validated
end to end: overlay-defined macros expand in interpret, compile AND self-host
modes (conformance 228/228 x3).
Moved comment and if-not; removed their Janet core-X fns + core-macro-names
entries. Note: Jolt defmacro is single-arity, so multi-arglist macros become one
arglist with & rest / destructuring (if-not uses [test then & [else]]). Macros
used by the compiler or earlier tiers (and/or/when/cond/case/cond->/->/declare/
doseq/when-let) stay in Janet until a load-order story exists for them.
full suite green, clojure-test-suite 3930.
Eighth pure-fn batch (jolt-1j0 phase 2), 2 leaf fns. dedupe is eager consecutive
dedup (no transducer arity, as before); seq-to-map-for-destructuring is the
internal &{:keys} helper. Largely exhausts the easy pure-eager tier.
conformance 228/228 x3, full suite green (incl. destructuring).
Sixth pure-fn batch (jolt-1j0 phase 2), 3 leaf fns, now with regression tests
per the per-batch workflow. reductions is the canonical form, so (reductions f [])
calls (f) -> [0] instead of the prior []; tree-seq is eager pre-order DFS.
conformance 228/228 x3, clojure-test-suite 3929, full suite green, bench A/B flat.
Two correctness bugs found while migrating:
- nthrest returned nil where Clojure returns () : for n>0 the walk yields
(seq xs) wrapped in (or ... ()), so an exhausted/nil walk is () not nil
(only n<=0 returns coll). Both the prior Janet impl and the first overlay
port got this wrong. nthrest.cljc 13/1/0 -> 14/0/0.
- if-let/when-let/if-some/when-some leaked their binding into the else branch:
they wrapped the whole if in (let* [name val] ...), so (let [x 5] (if-let
[x nil] x x)) returned nil instead of 5. Fixed to bind a fresh temp around the
if and rebind the name only inside the taken branch, matching Clojure.
conformance 218/218 x3, clojure-test-suite 3928 -> 3929.
Fifth pure-fn batch (jolt-1j0 phase 2), 6 leaf fns. nthrest uses the canonical
loop form (same neutral 13/1/0 on nthrest.cljc as the prior Janet version; the
1 fail is a pre-existing platform-conditional case). object?/undefined? are
always false on Jolt.
conformance 218/218 x3, clojure-test-suite 3928, core-bench A/B noise-only.
Fourth pure-fn batch (jolt-1j0 phase 2), 6 leaf fns. distinct? now uses
value-semantics (the prior Janet impl keyed a table by identity, so equal
collections compared distinct); nthnext uses the canonical loop form, which
fixes (nthnext nil nil) => nil and the nil-count cases the prior pre-check threw
on. replace preserves nil values via get-with-default.
conformance 218/218 x3, clojure-test-suite 3927 -> 3928 (canonical nthnext),
core-bench A/B flat. Per-batch gate caught a transient -1 from the first
nthnext rewrite; fixed before commit.
Second pure-fn batch (jolt-1j0 phase 2). All leaf fns; canonical Clojure defs.
split-with now returns seqs for the parts (drop-while is a seq), matching
Clojure rather than the prior all-vector result — value-equal either way.
conformance 218/218 x3, clojure-test-suite 3927, core-bench 2333ms (~baseline).
First pure-fn migration batch. Both are leaf fns (no internal Janet callers);
the Clojure defs (not . some / not . every?) match the prior Janet behavior.
Removed the core-* defns and their core-bindings entries.
conformance 218/218 x3, clojure-test-suite 3927, core-bench 2340ms vs 2336ms
baseline (noise).
Classify the 421 core-* fns into seed/macro/host/lazy/movable buckets and record
a compile-mode benchmark baseline to catch regressions as fns move from native
Janet to the self-hosted overlay. Worklist in jolt-core/clojure/core/MIGRATION.md.
Finding: after seed + host, the self-hosted compiler uses no clojure.core fns
beyond the existing kernel tier (+ atom/swap!/reset! host primitives), so the
compiler-dep kernel tier is already complete — confirmed by self-host
conformance passing.
def-node carried :meta nil when a def had no metadata, fn-node carried :name
nil for anonymous fns, analyze-arity carried :rest nil for fixed arities, and
empty-env carried :recur nil. These all mean genuine absence, and every
consumer reads an absent key back as nil — so writing the nil is gratuitous.
Dropping them keeps the common nodes (defs, fns, fixed arities) nil-free, so
once user maps preserve nil they stay plain structs instead of becoming phm.
Faithful nil values (a nil const :val) are left as-is; the back end handles
those. No-op today since structs already drop nil: suite, conformance 218/218
x3, fixpoint, fib all unchanged.
Three pure function combinators into the collection tier, as compositions of
mapv/every?/some/apply. Three more core-* primitives + table entries gone.
Battery 3916 -> 3920 (the canonical defs are more correct than the prior Janet
ones — some-fn returns the matching value, every-pred is properly boolean).
Conformance 218/218 all modes.
New collection tier (core/20-coll.clj): six pure, eager fns expressed as
compositions of frozen primitives (reduce/assoc/get/conj/filter/vec). Six more
core-* primitives + table entries gone from the Janet kernel.
Two semantics to preserve, caught by the suite:
- max-key/min-key use the canonical multi-arity form (strict </> on the first
pair, <=/>= in the fold) to reproduce the JVM IEEE-754 NaN behavior; a single
reduce got the NaN cases wrong.
- frequencies/group-by base on (hash-map), not the {} literal: a struct map
doesn't canonicalize collection keys across representations ({:a 1} literal vs
(hash-map :a 1)), a PHM does — same reason the Janet impl used make-phm.
Conformance 218/218 all modes; battery holds 3916.
Calls compile direct (target value embedded) iff the compiling unit has
direct-linking on, the target isn't ^:redef/^:dynamic, and it's an
already-defined fn; otherwise indirect (live var deref, redefinable). Direct
emission is guarded to function values only — embedding a non-function would make
Janet evaluate it as code.
:aot-core? (init opt, default true; JOLT_AOT_CORE=0 disables) compiles the core
tiers with direct-linking on so core->core calls are direct. User/REPL code
compiles indirect by default, so redefining functions in the REPL works with no
annotation, exactly like Clojure. :aot-core? false makes core indirect too — the
whole language becomes redefinable. ^:redef / ^:dynamic on a target force it
indirect even inside a direct-linked unit.
Also fixes a pre-existing gap this surfaced: compiled def dropped ALL var
metadata (so ^:dynamic was silently lost under compilation, and ^:redef couldn't
work). def metadata now flows analyzer -> IR -> back end (new form-sym-meta host
fn, def-node :meta, var-setter-meta) and is applied to the cell.
Matrix test in test/integration/direct-linking-test.janet. Conformance 218/218
in all three modes under both :aot-core? true and false; battery holds 3916;
binary rebuilt.
First fractal turn of the multi-stage bootstrap (jolt-tzo). The Clojure part of
clojure.core is now loaded in ordered tiers under jolt-core/clojure/core/. The
kernel tier (00-kernel: second/peek/subvec/mapv/update) holds the structural fns
the self-hosted compiler itself uses; in compile mode it is bootstrap-compiled
into clojure.core BEFORE the analyzer is built, so the analyzer binds those names
to the Clojure definitions instead of a not-yet-defined forward ref. That removes
the circularity that previously forced these to stay in Janet — the five core-*
primitives and their init-core! entries are gone.
Mechanism: backend/bootstrap-load-source (generalized from compile-load) builds a
source string into a target ns via the bootstrap; rebuild-compiler! recompiles
the self-hosted compiler against the current core (the rail for future turns,
exercised by the new fixpoint test). api/load-core-overlay! walks the ordered
tiers, bootstrap-loading kernel tiers and self-hosting the rest.
Also fixes a latent evaluator bug surfaced by the move: a fn rebinds current-ns
to its defining ns while it runs and restores on normal return, but a fn that
THREW unwound past its own restore, leaking the ns. try now restores the
try-entry ns on the catch/finally path, so referred-symbol resolution survives a
caught error (this was breaking is/testing in the suite harness after any thrown
assertion). Net battery +3 (3913 -> 3916); conformance 218/218 in all three
modes; binary builds and runs the embedded tiers.
Continue the kernel shrink. last/butlast become canonical Clojure defs
(pure first/next/loop/recur) in the core.clj overlay, dropping two
realize-for-iteration uses from the Janet kernel.
second was the obvious next candidate but can't move: the self-hosted
compiler front-end (analyzer.clj) calls it, and the compiler has to
compile the overlay, so anything it uses must already exist as a Janet
primitive. Documented that as a third clause of the safe-to-move rule.
Suite green, clojure-test-suite battery holds 3913 pass / 58 clean,
binary builds and runs the embedded overlay.
Phase 4 kernel-shrink seam. jolt-core/clojure/core.clj holds the Clojure portion
of clojure.core; api/init loads it into the clojure.core namespace after
init-core! interns the Janet primitives, routed through eval-toplevel so it
compiles via the self-hosted pipeline (or interprets when :compile? is off).
First three fns moved off the Janet side: ffirst, fnext, nnext (leaf, no internal
callers). Same results compiled or interpreted; full suite green; battery holds
3913.
eval-toplevel now routes every compiled form through backend/compile-and-eval
(analyzer -> IR -> Janet back end). The bootstrap compiler (compile-ast) stays
only to bootstrap jolt.ir/jolt.analyzer in backend/compile-load. Dropped the
:selfhost? flag and JOLT_SELFHOST env plumbing — self-host is the default.
Three correctness fixes the full-suite-under-self-host sweep exposed:
- analyzer/back end: recur into a variadic arity now compiles. The arity fn
gets an ordinary positional rest param (the collected seq), with a dispatch
wrapper, so (recur fixed... rest-seq) is a clean self-call like Clojure --
instead of punting to the interpreter, which hangs on this (jolt-4df).
- host_iface: interop heads (.foo/.-field/Foo.) and ns-management forms
(all-ns, create-ns, resolve, ...) now mark uncompilable so they fall back
instead of compiling to a bad call.
- back end: named-fn self-reference ((fn self [n] ... (self ...))) binds the
fn name via a var.
Full unit+integration suite green; clojure-test-suite battery holds 3913.
The self-hosted compiler now genuinely compiles to native Janet bytecode rather
than falling back to the interpreter:
- bootstrap fixes that let it compile the analyzer: (def x) with no init, and an
empty (do) (declare expands to one) no longer error;
- analyzer reordered so only the mutually-recursive is forward declared
(a deeply-nested forward ref to analyze-fn/analyze-try miscompiled to nil);
- back end emits native Janet ops for hot primitives (+,-,*,<,>,<=,>=,inc,dec)
on clojure.core refs, like the bootstrap's core-renames.
Result: self-hosted fib(30) runs in 0.52s (was ~50s interpreted). Full
clojure-test-suite via JOLT_SELFHOST runs compiled at 3869 pass (some compiled-
vs-interpreted divergences remain to chase toward full parity — jolt-4xc).
Default suite green; conformance 218/218; jank-conformance up to 135.
The portable analyzer now refers the host contract + IR ctors unqualified (host
form predicates renamed form-* to dodge core-renames), so the bootstrap compiles
it via its plain :var path — no qualified-ref compilation needed. ensure-analyzer
compile-loads jolt.ir + jolt.analyzer as native bytecode.
Result: the self-hosted pipeline (portable Clojure analyzer -> IR -> Janet back
end) runs the FULL clojure-test-suite at 3913 pass — parity with the interpreter
baseline — and fast (no timeouts), via JOLT_SELFHOST=1. Conformance 218/218.
Also fixes a real interpreter bug: multi-arity dispatch stored the variadic clause
by fixed-count (colliding) and only matched exact counts, so (f a b c..) on an
[a b & more] clause threw. Now the variadic clause dispatches for any count >= its
fixed arity.
Remaining (jolt-4xc): the compiled analyzer still errors analyzing a multi-arity
fn literal (falls back to the interpreter, now correct); compiling that is the
last coverage gap before flipping the self-hosted pipeline on by default.
Analyzer threads an env (locals + recur target) instead of a bare locals set, and
compiles loop*/recur, recur directly in a fixed-arity fn (each arity's name is its
recur target -> self-call), and try/catch/finally (-> Janet try + defer). recur
into a variadic arity isn't a recur target, so it falls back to the interpreter
rather than miscompiling the rest seq. Still 218/218 conformance via the pipeline,
now with these forms compiling natively instead of interpreting.
The Clojure analyzer + Janet back end now pass all 218 conformance cases (vs the
interpreter) through the self-hosted pipeline:
- host contract gains special? (the full special-form/uncompilable set), so the
analyzer falls back for forms it doesn't compile instead of miscompiling them
as calls;
- qualified symbol refs that don't resolve to a var (janet/…, Math/…, host
interop) and tagged literals (#"regex", #inst) and set literals throw
uncompilable -> interpreter;
- the back end's compile-and-eval is now hybrid: only the compile step is
guarded, runtime errors propagate.
Still interpreter-routed (coverage to grow next): loop/recur, recur in fn, try,
sets, destructuring.
The Clojure-in-Clojure front end now compiles and runs a real subset end to end:
arithmetic, if/do/let, vector/map literals, def + name-based global refs, fn,
defn (via macroexpand), recursion through the var cell, multi-arity, variadic,
and higher-order calls. Proven by test/integration/self-host-test.janet.
Pieces:
- jolt-core/jolt/analyzer.clj — PORTABLE Clojure analyzer: reader form -> IR,
depending only on the host contract (jolt.host), never on Janet.
- src/jolt/host_iface.janet — Janet impl of the jolt.host contract (form
introspection + resolve/macro/current-ns), installed into each ctx.
- src/jolt/backend.janet — Janet back end: IR -> Janet form -> eval; resolves
name-based :var nodes to cells and reuses runtime helpers.
Host Janet code lives in src/jolt/ (not a host/janet/ dir): Janet resolves
relative imports per file, so cross-dir ../../src/jolt/* imports load second
instances of compiler/types/core and corrupt state. The portability boundary is
the jolt.host namespace contract + jolt-core/, not the directory.
Notes: gensym in the back end must not be Janet's (shadowed by Jolt's via use);
a jolt bug — (into #{} coll) doesn't add elements — was worked around with reduce
conj in the analyzer (filed separately).
Adds doc/self-hosting-architecture.md: the portability design (portable front
end + host back end + host runtime, seam = a minimal host protocol + the IR),
grounded in the CLJS/Clojure-in-Clojure prior art. Decisions locked: minimal host
protocol as the seam; physical split with a jolt-core/ source root.
Adds jolt-core/ as a portable Clojure source root (dev path + embedded into the
binary alongside the rest of the stdlib) and jolt-core/jolt/ir.clj — host-neutral
IR node constructors (vars referenced by name, no host values embedded). Verified
it loads and runs under both interpreter and compiler.
