Fifth pure-fn batch (jolt-1j0 phase 2), 6 leaf fns. nthrest uses the canonical
loop form (same neutral 13/1/0 on nthrest.cljc as the prior Janet version; the
1 fail is a pre-existing platform-conditional case). object?/undefined? are
always false on Jolt.
conformance 218/218 x3, clojure-test-suite 3928, core-bench A/B noise-only.
Fourth pure-fn batch (jolt-1j0 phase 2), 6 leaf fns. distinct? now uses
value-semantics (the prior Janet impl keyed a table by identity, so equal
collections compared distinct); nthnext uses the canonical loop form, which
fixes (nthnext nil nil) => nil and the nil-count cases the prior pre-check threw
on. replace preserves nil values via get-with-default.
conformance 218/218 x3, clojure-test-suite 3927 -> 3928 (canonical nthnext),
core-bench A/B flat. Per-batch gate caught a transient -1 from the first
nthnext rewrite; fixed before commit.
Second pure-fn batch (jolt-1j0 phase 2). All leaf fns; canonical Clojure defs.
split-with now returns seqs for the parts (drop-while is a seq), matching
Clojure rather than the prior all-vector result — value-equal either way.
conformance 218/218 x3, clojure-test-suite 3927, core-bench 2333ms (~baseline).
First pure-fn migration batch. Both are leaf fns (no internal Janet callers);
the Clojure defs (not . some / not . every?) match the prior Janet behavior.
Removed the core-* defns and their core-bindings entries.
conformance 218/218 x3, clojure-test-suite 3927, core-bench 2340ms vs 2336ms
baseline (noise).
Classify the 421 core-* fns into seed/macro/host/lazy/movable buckets and record
a compile-mode benchmark baseline to catch regressions as fns move from native
Janet to the self-hosted overlay. Worklist in jolt-core/clojure/core/MIGRATION.md.
Finding: after seed + host, the self-hosted compiler uses no clojure.core fns
beyond the existing kernel tier (+ atom/swap!/reset! host primitives), so the
compiler-dep kernel tier is already complete — confirmed by self-host
conformance passing.
Add a maps-spec defspec covering nil values through the reader (literals), the
construction path, and the op surface (assoc/merge/merge-with/into/conj/zipmap/
select-keys/get-in/dissoc/reduce-kv, nil keys). Raise the clojure-test-suite
baseline 3919 -> 3926 to guard the assertions the fix unlocked.
With nil now preserved in maps, the ops that iterated via (keys (phm-to-struct
m)) or built struct/table results were still dropping nil-valued entries. Route
them through map-entries-of (all [k v] pairs incl nil-valued keys) and build
results via core-assoc / kvs->map / map-assoc1 (phm when a key/value is nil):
- merge, conj (map onto struct or phm), transient conj!
- merge-with: also use presence (contains?), not nil-of-value, to decide
whether to combine — a present nil triggers (f existing v), like Clojure
- reduce-kv: iterate phm-entries so nil-valued keys are visited
- select-keys / zipmap: collect entries, build via kvs->map
- get-in: walk with a sentinel so a present nil value isn't read as missing
conformance 218/218 x3, clojure.data/diff handles nil, suite green,
clojure-test-suite 3926 -> 3927.
Janet structs drop nil, so {:b nil}, (assoc {} :b nil) and friends silently
lost the entry — diverging from Clojure, where a nil value is still a present
key. Map construction now uses a phm (which preserves nil) whenever a key or
value is nil; the nil-free case stays a fast struct.
Touched the whole construction path: the reader (a nil-bearing map literal form
is a phm, not a struct), eval-form (evaluates phm map-literal forms, rebuilds
preserving nil), build-map-literal (compiled path), core-assoc (promotes on a
nil key/value), and the host map-form predicates h-map?/h-map-pairs (recognize
a phm form) so the self-hosted analyzer handles {:b nil} literals.
Safe now that the back end densifies phm IR nodes and the IR is nil-field-clean
(prior commits): const-nil nodes go phm but read back correctly, and the common
nodes stay structs, so fib stays compiled (0.08s, no interpret fallback).
Also fixes clojure.set/rename-keys, which deleted the old key via (assoc m old
nil) — a trick that only worked while nil was dropped; now uses dissoc.
conformance 218/218 x3, suite green, clojure-test-suite 3919 -> 3926.
def-node carried :meta nil when a def had no metadata, fn-node carried :name
nil for anonymous fns, analyze-arity carried :rest nil for fixed arities, and
empty-env carried :recur nil. These all mean genuine absence, and every
consumer reads an absent key back as nil — so writing the nil is gratuitous.
Dropping them keeps the common nodes (defs, fns, fixed arities) nil-free, so
once user maps preserve nil they stay plain structs instead of becoming phm.
Faithful nil values (a nil const :val) are left as-is; the back end handles
those. No-op today since structs already drop nil: suite, conformance 218/218
x3, fixpoint, fib all unchanged.
The back end reads IR node fields with raw (node :key) access, which assumes a
Janet struct. Once user maps preserve nil (coming), a node that carries a
nil-valued field — anonymous fn :name, nil const :val, def with no :meta, arity
with no :rest — is built as a phm instead, whose fields live under :buckets and
read back nil. norm-node densifies such a node via phm-to-struct, which drops
exactly those nil fields — precisely what the back end wants, since it already
treats an absent field as nil. Applied where a node first reaches the emitter
(emit entry, fn arities, invoke callee); structs pass through untouched.
No representation change yet, so this is a no-op today: full suite, conformance
218/218 x3, fixpoint, fib all unchanged. Keeps IR access a host-layer concern
so jolt.ir/jolt.analyzer stay portable.
phm-get returned the default for a key present with a nil value; fix it to
presence-check the bucket so a nil value reads back as nil, not the default.
core-keys/core-vals went through phm-to-struct, which drops nil-valued keys —
read phm-entries directly instead so those keys survive.
Isolated phm correctness only; no map-representation change. Battery holds at
3919, conformance 218/218 in all three modes. The earlier -8 came from
assoc's struct->phm promotion, not these accessors.
The back end emitted a direct Janet call (f args) for every :var/:local callee,
but Janet calling a pvec-table does get (no integer key -> nil), a keyword does
the wrong thing, etc. — so (let [v [1 2 3]] (v 0)), a keyword/meta-vector in a
binding, all returned nil in compile mode (interpret was fine). Now a direct call
is emitted only when the callee is provably a function: :fn / :host always, and a
:var whose current root is a function; everything else routes through jolt-call,
which dispatches IFn correctly (function fast-path first). User/core fn calls stay
direct, so no hot-path regression — fixpoint stage1==stage2==stage3 holds.
Conformance 218/218 all modes; full suite green. Spec cases added.
Document the two fixed behaviors in the contract spec: (with-meta x nil) clears
metadata, and a multi-collection map keeps nil elements (they're values, not
end-of-seq). The reader grammar (doc/grammar.ebnf) is unchanged — this session
touched semantics/stdlib, not surface syntax.
The reader yields set literals as a form ({:jolt/type :jolt/set ...}); edn now
walks the parsed result and builds actual sets (recursing into maps/vectors;
lists stay lists, never evaluated). Untagged-struct guard so symbols/chars/tagged
literals pass through. 47 -> 49 in the battery; only #uuid (no uuid type) remains.
clojure.edn/read-string now delegates to clojure.core/read-string (the
reader-based special form — parses, never evals) via a private helper, with the
opts-map :eof arity and nil/blank-input handling. 43 -> 47 in the stdlib battery.
This required an evaluator fix: special forms were matched by name only, so
clojure.edn/read-string dispatched the core read-string SPECIAL FORM instead of
the var. Now a head qualified to a non-core namespace resolves to its var; only
unqualified or clojure.core-qualified heads are special forms. Conformance
218/218 all modes; full suite green.
Remaining edn gaps (jolt-b7y): set literals read as a set FORM not a constructed
set, and #uuid has no real type.
The multi-coll map step detected end-of-seq with (nil? (ls-first cur)), which
also fired on a legitimate nil element — so mapping over a lazy-seq containing
nils truncated at the first nil. That's why (doall (map merge a b)) collapsed to
() as a reduce accumulator. Use seq-done? to detect exhaustion and push the value
(which may be nil). doall is unchanged; clojure.data restored to the canonical
(doall (map ...)) form. Conformance 218/218; battery 3919.
clojure.data: ported from the ClojureScript impl (jolt-native equality-partition
fn instead of host-type protocol dispatch). Verified against Clojure's own
data-test (12/13 canonical; the 13th needs nil-valued-map support, jolt-c7h).
Used mapv over the reference's (doall (map …)) — jolt's lazy multi-coll map +
doall don't force as a reduce accumulator (jolt-dzh).
clojure.zip: jolt's was a broken custom reimplementation; replaced with a port of
real clojure.zip (metadata-based locs). Uses (nth loc …) instead of (loc …)
because meta-bearing vectors aren't invocable as fns (jolt-vh5).
core: with-meta now accepts nil metadata (Clojure allows (with-meta x nil)); it
crashed calling keys on nil. This is what unblocked zip's make-node.
New vendored battery (test/clojure-stdlib/, from clojurust's suite with fixtures
corrected to match real Clojure): walk 34, zip 33, data 61 all clean; edn guarded
at 43 (still a stub — jolt-b7y). Conformance 218/218 all modes; full suite green.
The expensive multimethod path is the hierarchy fallback: when a dispatch value
isn't a direct method key, mm-fn walks every key with isa? (derive-based
dispatch). That resolution is now cached per dispatch value on the multimethod
var (:jolt/dispatch-cache), cleared in place by defmethod/prefer-method/
remove-method/remove-all-methods so a redefinition can't be served stale. Direct
(get methods dv) hits stay uncached — already a single lookup.
Test cases added to dispatch-cache-test (compile, interpret, aot-core off):
hierarchy hit is cached, a newly-added specific method is seen, removal re-exposes
the fallback. Conformance 218/218; full suite green.
Soundness gate for self-hosting (jolt-d0r). stage1 = analyzer built by the Janet
bootstrap; stage2 = analyzer rebuilt by compiling jolt.ir + jolt.analyzer through
stage1 (self-host); stage3 = the same self-rebuild again. A corpus of programs
must produce identical results across all three.
Compared BEHAVIORALLY rather than by emitted code: emitted Janet forms embed live
setter/getter closures (identity varies per compile) and the IR carries
representation-level gensyms and pvec internals, so structural equality fights the
representation. Behavioral parity is the property that actually matters and is
representation-independent.
Also corrects the battery baseline 3920 -> 3919: 3920 was a lucky run; 3919 is the
stable value (the 9 timeout-prone tests can shift the count by one).
Three pure function combinators into the collection tier, as compositions of
mapv/every?/some/apply. Three more core-* primitives + table entries gone.
Battery 3916 -> 3920 (the canonical defs are more correct than the prior Janet
ones — some-fn returns the matching value, every-pred is properly boolean).
Conformance 218/218 all modes.
New collection tier (core/20-coll.clj): six pure, eager fns expressed as
compositions of frozen primitives (reduce/assoc/get/conj/filter/vec). Six more
core-* primitives + table entries gone from the Janet kernel.
Two semantics to preserve, caught by the suite:
- max-key/min-key use the canonical multi-arity form (strict </> on the first
pair, <=/>= in the fold) to reproduce the JVM IEEE-754 NaN behavior; a single
reduce got the NaN cases wrong.
- frequencies/group-by base on (hash-map), not the {} literal: a struct map
doesn't canonicalize collection keys across representations ({:a 1} literal vs
(hash-map :a 1)), a PHM does — same reason the Janet impl used make-phm.
Conformance 218/218 all modes; battery holds 3916.
Host-value protocol dispatch (a protocol extended to Number/String/… called on a
host value) recomputed the candidate type-tag list and walked the registry — up
to ~15 map lookups plus an array alloc — on every call. It now caches
(most-specific-host-tag, protocol, method) -> fn, guarded by a registry
generation that bumps in register-protocol-method; re-extending a protocol
invalidates the cache. The single-lookup fast paths (deftype protocol dispatch,
direct multimethod dispatch) are untouched — they're already cheap, so no cache
overhead there. The multimethod hierarchy-fallback cache (narrower: only
derive-based dispatch) is deferred.
Test in test/integration/dispatch-cache-test.janet covers correctness under
re-extension in all three modes. Conformance 218/218; battery 3916.
var-get short-circuits to the root when no dynamic bindings are active (the
common case — the binding stack is only non-empty inside a `binding` block),
skipping the stack walk on every global deref.
The compiled indirect deref now emits (var-get (quote cell)) instead of routing
through a per-var memoized getter closure — one fewer call and no per-var closure
alloc. The cell is quoted so it embeds by reference: a bare table in argument
position is re-evaluated as a constructor, which deep-copies the cell (and any
atom in its root) on every call — that silently broke swap!/atoms/self-ref
lazy-seqs until the quote. Conformance 218/218 all three modes; battery 3916.
Calls compile direct (target value embedded) iff the compiling unit has
direct-linking on, the target isn't ^:redef/^:dynamic, and it's an
already-defined fn; otherwise indirect (live var deref, redefinable). Direct
emission is guarded to function values only — embedding a non-function would make
Janet evaluate it as code.
:aot-core? (init opt, default true; JOLT_AOT_CORE=0 disables) compiles the core
tiers with direct-linking on so core->core calls are direct. User/REPL code
compiles indirect by default, so redefining functions in the REPL works with no
annotation, exactly like Clojure. :aot-core? false makes core indirect too — the
whole language becomes redefinable. ^:redef / ^:dynamic on a target force it
indirect even inside a direct-linked unit.
Also fixes a pre-existing gap this surfaced: compiled def dropped ALL var
metadata (so ^:dynamic was silently lost under compilation, and ^:redef couldn't
work). def metadata now flows analyzer -> IR -> back end (new form-sym-meta host
fn, def-node :meta, var-setter-meta) and is applied to the cell.
Matrix test in test/integration/direct-linking-test.janet. Conformance 218/218
in all three modes under both :aot-core? true and false; battery holds 3916;
binary rebuilt.
Foundation for the linking-mode work (jolt-5ed). Every var cell carries a :gen
that bumps on bind-root / var-set(root) / alter-var-root — i.e. on redefinition.
Behavior-neutral on its own; it's the substrate the upcoming work needs:
direct-linked (sealed) call sites can detect staleness, and redefinition-aware
dispatch caches can invalidate per-var (only a redefined var's callers miss,
not a global flush). Full suite green, conformance 218/218 in all three modes.
First fractal turn of the multi-stage bootstrap (jolt-tzo). The Clojure part of
clojure.core is now loaded in ordered tiers under jolt-core/clojure/core/. The
kernel tier (00-kernel: second/peek/subvec/mapv/update) holds the structural fns
the self-hosted compiler itself uses; in compile mode it is bootstrap-compiled
into clojure.core BEFORE the analyzer is built, so the analyzer binds those names
to the Clojure definitions instead of a not-yet-defined forward ref. That removes
the circularity that previously forced these to stay in Janet — the five core-*
primitives and their init-core! entries are gone.
Mechanism: backend/bootstrap-load-source (generalized from compile-load) builds a
source string into a target ns via the bootstrap; rebuild-compiler! recompiles
the self-hosted compiler against the current core (the rail for future turns,
exercised by the new fixpoint test). api/load-core-overlay! walks the ordered
tiers, bootstrap-loading kernel tiers and self-hosting the rest.
Also fixes a latent evaluator bug surfaced by the move: a fn rebinds current-ns
to its defining ns while it runs and restores on normal return, but a fn that
THREW unwound past its own restore, leaking the ns. try now restores the
try-entry ns on the catch/finally path, so referred-symbol resolution survives a
caught error (this was breaking is/testing in the suite harness after any thrown
assertion). Net battery +3 (3913 -> 3916); conformance 218/218 in all three
modes; binary builds and runs the embedded tiers.
Continue the kernel shrink. last/butlast become canonical Clojure defs
(pure first/next/loop/recur) in the core.clj overlay, dropping two
realize-for-iteration uses from the Janet kernel.
second was the obvious next candidate but can't move: the self-hosted
compiler front-end (analyzer.clj) calls it, and the compiler has to
compile the overlay, so anything it uses must already exist as a Janet
primitive. Documented that as a third clause of the safe-to-move rule.
Suite green, clojure-test-suite battery holds 3913 pass / 58 clean,
binary builds and runs the embedded overlay.
Phase 4 kernel-shrink seam. jolt-core/clojure/core.clj holds the Clojure portion
of clojure.core; api/init loads it into the clojure.core namespace after
init-core! interns the Janet primitives, routed through eval-toplevel so it
compiles via the self-hosted pipeline (or interprets when :compile? is off).
First three fns moved off the Janet side: ffirst, fnext, nnext (leaf, no internal
callers). Same results compiled or interpreted; full suite green; battery holds
3913.
compiled?/get-compiled-forms/clear-compiled-cache were never called, and
load-ns only ever wrote to :compiled-cache with no reader. All dead once the
self-host path landed. Removed the helpers, the cache writes, and the
:compiled-cache env field.
eval-toplevel now routes every compiled form through backend/compile-and-eval
(analyzer -> IR -> Janet back end). The bootstrap compiler (compile-ast) stays
only to bootstrap jolt.ir/jolt.analyzer in backend/compile-load. Dropped the
:selfhost? flag and JOLT_SELFHOST env plumbing — self-host is the default.
Three correctness fixes the full-suite-under-self-host sweep exposed:
- analyzer/back end: recur into a variadic arity now compiles. The arity fn
gets an ordinary positional rest param (the collected seq), with a dispatch
wrapper, so (recur fixed... rest-seq) is a clean self-call like Clojure --
instead of punting to the interpreter, which hangs on this (jolt-4df).
- host_iface: interop heads (.foo/.-field/Foo.) and ns-management forms
(all-ns, create-ns, resolve, ...) now mark uncompilable so they fall back
instead of compiling to a bad call.
- back end: named-fn self-reference ((fn self [n] ... (self ...))) binds the
fn name via a var.
Full unit+integration suite green; clojure-test-suite battery holds 3913.
conformance-test now runs every case under interpret, the bootstrap compiler, AND
the self-hosted pipeline (portable Clojure analyzer -> IR -> Janet back end). All
three pass 218/218, so the self-hosted compiler can't silently regress in CI.
The self-hosted compiler now genuinely compiles to native Janet bytecode rather
than falling back to the interpreter:
- bootstrap fixes that let it compile the analyzer: (def x) with no init, and an
empty (do) (declare expands to one) no longer error;
- analyzer reordered so only the mutually-recursive is forward declared
(a deeply-nested forward ref to analyze-fn/analyze-try miscompiled to nil);
- back end emits native Janet ops for hot primitives (+,-,*,<,>,<=,>=,inc,dec)
on clojure.core refs, like the bootstrap's core-renames.
Result: self-hosted fib(30) runs in 0.52s (was ~50s interpreted). Full
clojure-test-suite via JOLT_SELFHOST runs compiled at 3869 pass (some compiled-
vs-interpreted divergences remain to chase toward full parity — jolt-4xc).
Default suite green; conformance 218/218; jank-conformance up to 135.
The portable analyzer now refers the host contract + IR ctors unqualified (host
form predicates renamed form-* to dodge core-renames), so the bootstrap compiles
it via its plain :var path — no qualified-ref compilation needed. ensure-analyzer
compile-loads jolt.ir + jolt.analyzer as native bytecode.
Result: the self-hosted pipeline (portable Clojure analyzer -> IR -> Janet back
end) runs the FULL clojure-test-suite at 3913 pass — parity with the interpreter
baseline — and fast (no timeouts), via JOLT_SELFHOST=1. Conformance 218/218.
Also fixes a real interpreter bug: multi-arity dispatch stored the variadic clause
by fixed-count (colliding) and only matched exact counts, so (f a b c..) on an
[a b & more] clause threw. Now the variadic clause dispatches for any count >= its
fixed arity.
Remaining (jolt-4xc): the compiled analyzer still errors analyzing a multi-arity
fn literal (falls back to the interpreter, now correct); compiling that is the
last coverage gap before flipping the self-hosted pipeline on by default.
- compiler/resolve-macro now resolves :as aliases (like resolve-var), so aliased
macro calls (t/is) are recognized and compiled via expansion instead of falling
back. Compile-mode suite holds at 3932.
- host/current-ns reads a backend-stashed :compile-ns rather than ctx-current-ns:
the interpreter rebinds current-ns to a fn's defining ns while it runs, so the
interpreted analyzer (defined in jolt.analyzer) would otherwise mis-resolve the
user's compile ns. Fixes def/intern targeting the wrong namespace.
Analyzer stays interpreted (compiling it needs qualified-ref compilation, which
regresses the clojure.test shim — jolt-4xc). Self-host pipeline correct (218/218),
full suite green.
core-let now expands destructuring (Clojure's destructure algorithm: nth/nthnext/
get over gensym roots) so both the interpreter and the compiler see only plain
bindings — the compiler compiles destructuring for free, and it's a step toward
moving destructuring out of the kernel. Handles vector (& rest, :as, nested), map
(:keys/:strs/:syms incl. namespaced, :as, :or, explicit pairs).
Two real bugs fixed in passing:
- compiler let*/loop* analyzed all binding inits in the OUTER scope, so a later
binding couldn't see an earlier one ((let [x 1 y x] y) miscompiled). Now scope
accumulates across bindings.
- core-nth treated an explicit nil not-found as 'no default' and threw on out of
bounds; (nth coll i nil) now returns nil (Clojure semantics), via arity.
Baselines held: conformance 218/218 both modes, clojure-test-suite 3913 interp /
3932 compile, destructuring-spec green.
suite-worker honors JOLT_SELFHOST=1 to route the clojure-test-suite through the
self-hosted pipeline (for validating once the analyzer compiles).
The analyzer stays interpreted: compiling it via the bootstrap miscompiles
(qualified refs regressed compile mode 3932->1181; compile-loading yields
corrupted IR) — tracked in jolt-4xc. Correctness is unaffected (the self-hosted
pipeline passes 218/218 conformance); only speed is pending.
Analyzer threads an env (locals + recur target) instead of a bare locals set, and
compiles loop*/recur, recur directly in a fixed-arity fn (each arity's name is its
recur target -> self-call), and try/catch/finally (-> Janet try + defer). recur
into a variadic arity isn't a recur target, so it falls back to the interpreter
rather than miscompiling the rest seq. Still 218/218 conformance via the pipeline,
now with these forms compiling natively instead of interpreting.
The Clojure analyzer + Janet back end now pass all 218 conformance cases (vs the
interpreter) through the self-hosted pipeline:
- host contract gains special? (the full special-form/uncompilable set), so the
analyzer falls back for forms it doesn't compile instead of miscompiling them
as calls;
- qualified symbol refs that don't resolve to a var (janet/…, Math/…, host
interop) and tagged literals (#"regex", #inst) and set literals throw
uncompilable -> interpreter;
- the back end's compile-and-eval is now hybrid: only the compile step is
guarded, runtime errors propagate.
Still interpreter-routed (coverage to grow next): loop/recur, recur in fn, try,
sets, destructuring.
The Clojure-in-Clojure front end now compiles and runs a real subset end to end:
arithmetic, if/do/let, vector/map literals, def + name-based global refs, fn,
defn (via macroexpand), recursion through the var cell, multi-arity, variadic,
and higher-order calls. Proven by test/integration/self-host-test.janet.
Pieces:
- jolt-core/jolt/analyzer.clj — PORTABLE Clojure analyzer: reader form -> IR,
depending only on the host contract (jolt.host), never on Janet.
- src/jolt/host_iface.janet — Janet impl of the jolt.host contract (form
introspection + resolve/macro/current-ns), installed into each ctx.
- src/jolt/backend.janet — Janet back end: IR -> Janet form -> eval; resolves
name-based :var nodes to cells and reuses runtime helpers.
Host Janet code lives in src/jolt/ (not a host/janet/ dir): Janet resolves
relative imports per file, so cross-dir ../../src/jolt/* imports load second
instances of compiler/types/core and corrupt state. The portability boundary is
the jolt.host namespace contract + jolt-core/, not the directory.
Notes: gensym in the back end must not be Janet's (shadowed by Jolt's via use);
a jolt bug — (into #{} coll) doesn't add elements — was worked around with reduce
conj in the analyzer (filed separately).
Adds doc/self-hosting-architecture.md: the portability design (portable front
end + host back end + host runtime, seam = a minimal host protocol + the IR),
grounded in the CLJS/Clojure-in-Clojure prior art. Decisions locked: minimal host
protocol as the seam; physical split with a jolt-core/ source root.
Adds jolt-core/ as a portable Clojure source root (dev path + embedded into the
binary alongside the rest of the stdlib) and jolt-core/jolt/ir.clj — host-neutral
IR node constructors (vars referenced by name, no host values embedded). Verified
it loads and runs under both interpreter and compiler.
README and the self-hosting design doc now describe the current pipeline:
compile-by-default with an always-correct interpreter fallback, var-cell late
binding, parity validation, and AOT images. The staged path marks
var-indirection, hybrid+coverage, and compile-by-default+AOT done; self-hosting
the compiler and moving core to Clojure remain.
Adds src/jolt/aot.janet: save-ns / load-ns-image marshal a namespace's compiled
var cells to a Janet bytecode image and load them back, skipping
parse/analyze/emit/compile on reload.
Compiled functions close over core fns (cfunctions) that can't marshal by value,
so we marshal against a dictionary built from the baked-in runtime env
(env-lookup jolt-runtime-env, which chains to the Janet boot env): core fns and
builtins are referenced by name, only user bytecode and its var cells are
serialized. The runtime env is identical at save and load time (same binary), so
the dictionaries match.
Test round-trips a namespace (constant, fn over it, core-fn user, recursion) into
a fresh context and confirms the loaded vars run and stay redefinable.
The jolt binary now compiles each form to Janet bytecode by default (REPL, file,
-e, -m, nrepl). The hybrid path keeps it correct — forms the compiler can't
handle fall back to the interpreter — and it's validated at parity with the
interpreter across the conformance suite (218/218) and the clojure-test-suite
(3932 pass / 124 error vs 3913 / 125). JOLT_INTERPRET=1 forces the interpreter.
analyze-form now mirrors resolve-sym's full resolution order: jolt var (current
ns / clojure.core), then the runtime/Janet env, then a pending forward-ref cell.
Previously a name that resolves only via the Janet-env fallback (int?, type, …)
interned an empty var and miscompiled; now it emits the runtime binding directly,
matching the interpreter.
suite-worker honors JOLT_COMPILE=1 to run the whole clojure-test-suite through
the compile path. Under compilation it now passes 3932 / errors 124 — at parity
with the interpreter baseline (3913 / 125) across 4600+ assertions, confirming
the hybrid compile path doesn't diverge from the interpreter.
Running the conformance suite under compile mode surfaced many forms that
silently miscompiled (the hybrid fallback only catches compile-time errors,
not wrong results). Fixes:
- Global var resolution now mirrors the interpreter's resolve-var: current ns
(which holds refers) then clojure.core, instead of interning an empty var in
the current ns. This was the big one — every core fn not in core-renames
(iterate, update, subvec, reductions, ...) derefed to nil from a user ns.
- Map literals evaluate their keys and values (were emitted as quoted data);
build via build-map-literal, mirroring the interpreter (struct, or phm when a
key is a collection).
- Vector literals build a mode-appropriate jolt vector via make-vec (pvec when
immutable, array when mutable) instead of a bare Janet tuple, so compiled and
interpreted vectors share one representation (type-strict ops like rseq
rejected tuples).
- Core fn values resolve dynamically from the runtime env instead of a
hand-maintained table that had drifted: core-apply mapped to Janet's native
apply (rejects pvec tails), core-some mapped to core-some?. Removed the table
and the bogus some rename.
- analyze-form throws uncompilable on interpreter special forms it doesn't
implement and on definitional/host macros (deftype, defprotocol, reify,
binding, letfn, read-string, regex/tagged literals, ...), so they fall back
to the interpreter instead of miscompiling — including nested in compiled
forms.
conformance-test.janet now runs every case under both interpreter and compiler
so compile-mode correctness is guarded in CI.
Moves the per-form routing (compile when :compile?, stateful forms always
interpret, interpreter fallback for uncompilable forms) into loader as
eval-toplevel, used by both load-ns and api/eval-one. Previously load-ns's
compile path called compile-and-eval directly with no fallback and no
stateful-form handling, so it would have broken ns/require/defmacro under
compile-by-default.
Array forms whose head is not a symbol (e.g. ((fn ...) args)) compile like
any other call, matching the prior eval-one behavior.
Rewrites fn* analysis/emit. Each arity compiles to a named Janet fn whose
name is its recur target, so recur is a self-call (Janet tail-calls it) —
this fixes recur used directly inside a fn, which previously miscompiled to
a runtime error that the hybrid fallback couldn't catch.
Multi-arity fns dispatch on arg count: fixed arities match exactly, the
variadic arity matches >= its fixed count. The rest param is an ordinary
param holding a seq, so (recur fixed... rest-seq) into a variadic arity
works as in Clojure. Single fixed-arity fns keep the direct, dispatch-free
shape so the hot path stays fast.
Destructuring params still route to the interpreter via uncompilable.
Tests cover named recursion, multi-arity (incl. variadic clause), recur in
fn, and recur into a variadic arity.
