Commit graph

267 commits

Author SHA1 Message Date
Yogthos
24b217d314 core: move cond->>/assert/delay/future/letfn macros to overlay
Phase 3 batch 4 (jolt-1j0), 5 macros. cond->> (thread-last) is safe; cond->
stays (compiler uses it). delay/future expand to make-delay/future-call host
fns; letfn builds its fn* binding via a template (cons/list in a macro body make
a plist the evaluator can't call as a form).

conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
2026-06-07 01:04:21 -04:00
Yogthos
31a257ce70 core: move as->/some->/some->>/doto/when-first macros to overlay
Phase 3 batch 3 (jolt-1j0), 5 macros. Establishes the macro-body toolkit:
fresh-sym = (symbol (str (gensym))) for a shared explicit jolt symbol (a bare
(gensym) returns a Janet symbol the destructurer rejects), and splicing binding
pairs into a TEMPLATE vector so core-let sees a tuple form, not a runtime pvec.

when-first stays first-based (lazy-safe): Clojure's (seq coll) form realizes an
infinite coll like (repeat nil) under Jolt's eager evaluator and hangs — the
per-batch battery caught it (when_first.cljc). Clojure-correct seq form waits on
Phase 5 laziness.

conformance 228/228 x3, clojure-test-suite 3930, full suite green, bench flat.
2026-06-07 00:55:19 -04:00
Yogthos
e611034550 core: move if-let/if-some/when-some/while/dotimes macros to overlay
Phase 3 batch 2 (jolt-1j0), 5 user-facing macros as defmacro + syntax-quote.
The conditional-binding macros use the auto-gensym temp# idiom so the name binds
only in the taken branch (the else/empty branch sees the outer scope, carrying
forward the earlier scope fix). when-let stays in Janet for now — 20-coll uses
it, so it must remain available before the macro tier loads.

Note: gensym in a macro body resolves to Janet's 0-arity builtin, so explicit
(gensym "prefix") fails — use template auto-gensym (foo#) instead.

conformance 228/228 x3, full suite green (incl. the if-let scope regression spec,
now exercising the overlay macro), clojure-test-suite 3930, bench A/B flat.
2026-06-07 00:38:02 -04:00
Yogthos
330a3a23d9 core: start macro tier — move comment/if-not to the Clojure overlay (jolt-1j0 phase 3)
New 30-macros tier (registered after 20-coll) for user-facing macros expressed as
defmacro + syntax-quote instead of hand-built Janet form-transformers. Validated
end to end: overlay-defined macros expand in interpret, compile AND self-host
modes (conformance 228/228 x3).

Moved comment and if-not; removed their Janet core-X fns + core-macro-names
entries. Note: Jolt defmacro is single-arity, so multi-arglist macros become one
arglist with & rest / destructuring (if-not uses [test then & [else]]). Macros
used by the compiler or earlier tiers (and/or/when/cond/case/cond->/->/declare/
doseq/when-let) stay in Janet until a load-order story exists for them.

full suite green, clojure-test-suite 3930.
2026-06-07 00:29:34 -04:00
Yogthos
f0e111563b core: move dedupe + seq-to-map-for-destructuring to overlay (+ tests)
Eighth pure-fn batch (jolt-1j0 phase 2), 2 leaf fns. dedupe is eager consecutive
dedup (no transducer arity, as before); seq-to-map-for-destructuring is the
internal &{:keys} helper. Largely exhausts the easy pure-eager tier.

conformance 228/228 x3, full suite green (incl. destructuring).
2026-06-07 00:14:40 -04:00
Yogthos
fec5a10ccf core: move some/flatten/interleave/rationalize to overlay (+ spec tests)
Seventh pure-fn batch (jolt-1j0 phase 2), 4 leaf fns. flatten is the canonical
tree-seq form, which also flattens lists (sequential?) — the prior Janet impl's
seqish? predicate missed them. some/interleave eager; rationalize identity (no
ratio type).

conformance 228/228 x3, clojure-test-suite 3929 -> 3930 (flatten lists fix),
full suite green, bench A/B flat.
2026-06-07 00:09:27 -04:00
Yogthos
921b395dfd core: move comparator/reductions/tree-seq to overlay (+ spec tests)
Sixth pure-fn batch (jolt-1j0 phase 2), 3 leaf fns, now with regression tests
per the per-batch workflow. reductions is the canonical form, so (reductions f [])
calls (f) -> [0] instead of the prior []; tree-seq is eager pre-order DFS.

conformance 228/228 x3, clojure-test-suite 3929, full suite green, bench A/B flat.
2026-06-06 23:58:11 -04:00
Yogthos
c35cf7bdbc migration doc: make regression tests part of the per-batch workflow 2026-06-06 23:51:40 -04:00
Yogthos
e646af123a test: regression coverage for the core migration + the two bug fixes
Spec suites (interpret) for the overlay-migrated fns and both fixes:
  - control-flow: if-let/when-let/if-some/when-some else-scope (9 cases)
  - predicates: not-any?/idents/numeric preds/NaN?/abs/object?/... (24 cases)
  - sequences: nthrest()/nthnext/distinct? value-eq/replace nil/take-last/... (23)
Plus 10 cases added to the all-3-modes conformance set for the highest-risk
fixes (if-let scope, nthrest (), distinct? value-eq), so compile + self-host are
guarded too: conformance 218 -> 228 in all three modes.

Full suite green.
2026-06-06 23:51:13 -04:00
Yogthos
f22eb660d3 fix nthrest empty-list semantics + if-let/when-let/if-some/when-some scoping
Two correctness bugs found while migrating:

- nthrest returned nil where Clojure returns () : for n>0 the walk yields
  (seq xs) wrapped in (or ... ()), so an exhausted/nil walk is () not nil
  (only n<=0 returns coll). Both the prior Janet impl and the first overlay
  port got this wrong. nthrest.cljc 13/1/0 -> 14/0/0.

- if-let/when-let/if-some/when-some leaked their binding into the else branch:
  they wrapped the whole if in (let* [name val] ...), so (let [x 5] (if-let
  [x nil] x x)) returned nil instead of 5. Fixed to bind a fresh temp around the
  if and rebind the name only inside the taken branch, matching Clojure.

conformance 218/218 x3, clojure-test-suite 3928 -> 3929.
2026-06-06 23:42:08 -04:00
Yogthos
185cf30c1d core: move nthrest/abs/NaN?/object?/undefined?/keyword-identical? to overlay
Fifth pure-fn batch (jolt-1j0 phase 2), 6 leaf fns. nthrest uses the canonical
loop form (same neutral 13/1/0 on nthrest.cljc as the prior Janet version; the
1 fail is a pre-existing platform-conditional case). object?/undefined? are
always false on Jolt.

conformance 218/218 x3, clojure-test-suite 3928, core-bench A/B noise-only.
2026-06-06 23:34:31 -04:00
Yogthos
4aec21e603 core: move distinct?/replace/nthnext/bounded-count/run!/completing to overlay
Fourth pure-fn batch (jolt-1j0 phase 2), 6 leaf fns. distinct? now uses
value-semantics (the prior Janet impl keyed a table by identity, so equal
collections compared distinct); nthnext uses the canonical loop form, which
fixes (nthnext nil nil) => nil and the nil-count cases the prior pre-check threw
on. replace preserves nil values via get-with-default.

conformance 218/218 x3, clojure-test-suite 3927 -> 3928 (canonical nthnext),
core-bench A/B flat. Per-batch gate caught a transient -1 from the first
nthnext rewrite; fixed before commit.
2026-06-06 23:26:36 -04:00
Yogthos
b1daa4bcbd core: move numeric predicates + replicate/take-last/drop-last to overlay
Third pure-fn batch (jolt-1j0 phase 2), 9 leaf fns: ratio?/decimal? (always
false on Jolt), rational?/nat-int?/neg-int?/pos-int? (reduce to int?), replicate,
take-last, drop-last. Canonical Clojure defs over already-frozen primitives.

conformance 218/218 x3, clojure-test-suite 3927. core-bench is load-sensitive in
absolute terms; A/B under identical load shows this batch == prior (no regression).
2026-06-06 23:14:49 -04:00
Yogthos
681b007b7a core: move split-at/split-with/ident?/qualified-ident?/simple-ident? to overlay
Second pure-fn batch (jolt-1j0 phase 2). All leaf fns; canonical Clojure defs.
split-with now returns seqs for the parts (drop-while is a seq), matching
Clojure rather than the prior all-vector result — value-equal either way.

conformance 218/218 x3, clojure-test-suite 3927, core-bench 2333ms (~baseline).
2026-06-06 23:05:24 -04:00
Yogthos
16cd617bd6 core: move not-any?/not-every? to the Clojure overlay (jolt-1j0 phase 2)
First pure-fn migration batch. Both are leaf fns (no internal Janet callers);
the Clojure defs (not . some / not . every?) match the prior Janet behavior.
Removed the core-* defns and their core-bindings entries.

conformance 218/218 x3, clojure-test-suite 3927, core-bench 2340ms vs 2336ms
baseline (noise).
2026-06-06 23:01:25 -04:00
Yogthos
6282ef0a39 core migration: phase 0 audit + perf baseline (jolt-1j0)
Classify the 421 core-* fns into seed/macro/host/lazy/movable buckets and record
a compile-mode benchmark baseline to catch regressions as fns move from native
Janet to the self-hosted overlay. Worklist in jolt-core/clojure/core/MIGRATION.md.

Finding: after seed + host, the self-hosted compiler uses no clojure.core fns
beyond the existing kernel tier (+ atom/swap!/reset! host primitives), so the
compiler-dep kernel tier is already complete — confirmed by self-host
conformance passing.
2026-06-06 22:54:07 -04:00
Yogthos
14d3cb1de4 test: lock in nil-valued map preservation (jolt-c7h)
Add a maps-spec defspec covering nil values through the reader (literals), the
construction path, and the op surface (assoc/merge/merge-with/into/conj/zipmap/
select-keys/get-in/dissoc/reduce-kv, nil keys). Raise the clojure-test-suite
baseline 3919 -> 3926 to guard the assertions the fix unlocked.
2026-06-06 22:35:26 -04:00
Yogthos
c43f261993 core: nil-correct the rest of the map ops
With nil now preserved in maps, the ops that iterated via (keys (phm-to-struct
m)) or built struct/table results were still dropping nil-valued entries. Route
them through map-entries-of (all [k v] pairs incl nil-valued keys) and build
results via core-assoc / kvs->map / map-assoc1 (phm when a key/value is nil):

  - merge, conj (map onto struct or phm), transient conj!
  - merge-with: also use presence (contains?), not nil-of-value, to decide
    whether to combine — a present nil triggers (f existing v), like Clojure
  - reduce-kv: iterate phm-entries so nil-valued keys are visited
  - select-keys / zipmap: collect entries, build via kvs->map
  - get-in: walk with a sentinel so a present nil value isn't read as missing

conformance 218/218 x3, clojure.data/diff handles nil, suite green,
clojure-test-suite 3926 -> 3927.
2026-06-06 22:29:55 -04:00
Yogthos
c280b10b29 maps: preserve nil keys/values (jolt-c7h)
Janet structs drop nil, so {:b nil}, (assoc {} :b nil) and friends silently
lost the entry — diverging from Clojure, where a nil value is still a present
key. Map construction now uses a phm (which preserves nil) whenever a key or
value is nil; the nil-free case stays a fast struct.

Touched the whole construction path: the reader (a nil-bearing map literal form
is a phm, not a struct), eval-form (evaluates phm map-literal forms, rebuilds
preserving nil), build-map-literal (compiled path), core-assoc (promotes on a
nil key/value), and the host map-form predicates h-map?/h-map-pairs (recognize
a phm form) so the self-hosted analyzer handles {:b nil} literals.

Safe now that the back end densifies phm IR nodes and the IR is nil-field-clean
(prior commits): const-nil nodes go phm but read back correctly, and the common
nodes stay structs, so fib stays compiled (0.08s, no interpret fallback).

Also fixes clojure.set/rename-keys, which deleted the old key via (assoc m old
nil) — a trick that only worked while nil was dropped; now uses dissoc.

conformance 218/218 x3, suite green, clojure-test-suite 3919 -> 3926.
2026-06-06 22:19:35 -04:00
Yogthos
b2ff65972b ir/analyzer: omit absent fields instead of writing nil
def-node carried :meta nil when a def had no metadata, fn-node carried :name
nil for anonymous fns, analyze-arity carried :rest nil for fixed arities, and
empty-env carried :recur nil. These all mean genuine absence, and every
consumer reads an absent key back as nil — so writing the nil is gratuitous.

Dropping them keeps the common nodes (defs, fns, fixed arities) nil-free, so
once user maps preserve nil they stay plain structs instead of becoming phm.
Faithful nil values (a nil const :val) are left as-is; the back end handles
those. No-op today since structs already drop nil: suite, conformance 218/218
x3, fixpoint, fib all unchanged.
2026-06-06 22:05:52 -04:00
Yogthos
22ff737f8b backend: densify phm IR nodes to structs at the emitter boundary
The back end reads IR node fields with raw (node :key) access, which assumes a
Janet struct. Once user maps preserve nil (coming), a node that carries a
nil-valued field — anonymous fn :name, nil const :val, def with no :meta, arity
with no :rest — is built as a phm instead, whose fields live under :buckets and
read back nil. norm-node densifies such a node via phm-to-struct, which drops
exactly those nil fields — precisely what the back end wants, since it already
treats an absent field as nil. Applied where a node first reaches the emitter
(emit entry, fn arities, invoke callee); structs pass through untouched.

No representation change yet, so this is a no-op today: full suite, conformance
218/218 x3, fixpoint, fib all unchanged. Keeps IR access a host-layer concern
so jolt.ir/jolt.analyzer stay portable.
2026-06-06 22:00:21 -04:00
Yogthos
bd76eddb74 phm: preserve nil-valued keys in get/keys/vals
phm-get returned the default for a key present with a nil value; fix it to
presence-check the bucket so a nil value reads back as nil, not the default.
core-keys/core-vals went through phm-to-struct, which drops nil-valued keys —
read phm-entries directly instead so those keys survive.

Isolated phm correctness only; no map-representation change. Battery holds at
3919, conformance 218/218 in all three modes. The earlier -8 came from
assoc's struct->phm promotion, not these accessors.
2026-06-06 21:54:07 -04:00
Yogthos
0878c803d7 compiler: IFn collections held in a var/local are callable (jolt-vh5)
The back end emitted a direct Janet call (f args) for every :var/:local callee,
but Janet calling a pvec-table does get (no integer key -> nil), a keyword does
the wrong thing, etc. — so (let [v [1 2 3]] (v 0)), a keyword/meta-vector in a
binding, all returned nil in compile mode (interpret was fine). Now a direct call
is emitted only when the callee is provably a function: :fn / :host always, and a
:var whose current root is a function; everything else routes through jolt-call,
which dispatches IFn correctly (function fast-path first). User/core fn calls stay
direct, so no hot-path regression — fixpoint stage1==stage2==stage3 holds.
Conformance 218/218 all modes; full suite green. Spec cases added.
2026-06-06 20:03:49 -04:00
Yogthos
d189ee2ded spec: regression cases for with-meta nil and multi-coll map nil elements
Document the two fixed behaviors in the contract spec: (with-meta x nil) clears
metadata, and a multi-collection map keeps nil elements (they're values, not
end-of-seq). The reader grammar (doc/grammar.ebnf) is unchanged — this session
touched semantics/stdlib, not surface syntax.
2026-06-06 19:52:54 -04:00
Yogthos
711a938943 edn: construct set/nested values from reader forms (49/1, only #uuid left)
The reader yields set literals as a form ({:jolt/type :jolt/set ...}); edn now
walks the parsed result and builds actual sets (recursing into maps/vectors;
lists stay lists, never evaluated). Untagged-struct guard so symbols/chars/tagged
literals pass through. 47 -> 49 in the battery; only #uuid (no uuid type) remains.
2026-06-06 19:49:07 -04:00
Yogthos
1bd676c242 edn: port read-string onto the reader; fix special-form ns shadowing
clojure.edn/read-string now delegates to clojure.core/read-string (the
reader-based special form — parses, never evals) via a private helper, with the
opts-map :eof arity and nil/blank-input handling. 43 -> 47 in the stdlib battery.

This required an evaluator fix: special forms were matched by name only, so
clojure.edn/read-string dispatched the core read-string SPECIAL FORM instead of
the var. Now a head qualified to a non-core namespace resolves to its var; only
unqualified or clojure.core-qualified heads are special forms. Conformance
218/218 all modes; full suite green.

Remaining edn gaps (jolt-b7y): set literals read as a set FORM not a constructed
set, and #uuid has no real type.
2026-06-06 19:45:30 -04:00
Yogthos
fc88ecdc74 data: keep mapv (vector result) — list-with-nil destructures wrong positionally 2026-06-06 19:30:20 -04:00
Yogthos
bfb49f23ab core: multi-collection map handles nil elements in a lazy-seq input (jolt-dzh)
The multi-coll map step detected end-of-seq with (nil? (ls-first cur)), which
also fired on a legitimate nil element — so mapping over a lazy-seq containing
nils truncated at the first nil. That's why (doall (map merge a b)) collapsed to
() as a reduce accumulator. Use seq-done? to detect exhaustion and push the value
(which may be nil). doall is unchanged; clojure.data restored to the canonical
(doall (map ...)) form. Conformance 218/218; battery 3919.
2026-06-06 19:28:47 -04:00
Yogthos
e623968b45 stdlib: port clojure.data, rewrite clojure.zip; fix with-meta nil; add battery
clojure.data: ported from the ClojureScript impl (jolt-native equality-partition
fn instead of host-type protocol dispatch). Verified against Clojure's own
data-test (12/13 canonical; the 13th needs nil-valued-map support, jolt-c7h).
Used mapv over the reference's (doall (map …)) — jolt's lazy multi-coll map +
doall don't force as a reduce accumulator (jolt-dzh).

clojure.zip: jolt's was a broken custom reimplementation; replaced with a port of
real clojure.zip (metadata-based locs). Uses (nth loc …) instead of (loc …)
because meta-bearing vectors aren't invocable as fns (jolt-vh5).

core: with-meta now accepts nil metadata (Clojure allows (with-meta x nil)); it
crashed calling keys on nil. This is what unblocked zip's make-node.

New vendored battery (test/clojure-stdlib/, from clojurust's suite with fixtures
corrected to match real Clojure): walk 34, zip 33, data 61 all clean; edn guarded
at 43 (still a stub — jolt-b7y). Conformance 218/218 all modes; full suite green.
2026-06-06 19:02:26 -04:00
Yogthos
c975f5d1c3 perf: generation-guarded cache for multimethod hierarchy dispatch
The expensive multimethod path is the hierarchy fallback: when a dispatch value
isn't a direct method key, mm-fn walks every key with isa? (derive-based
dispatch). That resolution is now cached per dispatch value on the multimethod
var (:jolt/dispatch-cache), cleared in place by defmethod/prefer-method/
remove-method/remove-all-methods so a redefinition can't be served stale. Direct
(get methods dv) hits stay uncached — already a single lookup.

Test cases added to dispatch-cache-test (compile, interpret, aot-core off):
hierarchy hit is cached, a newly-added specific method is seen, removal re-exposes
the fallback. Conformance 218/218; full suite green.
2026-06-06 18:36:11 -04:00
Yogthos
e02ccab4c0 test: bootstrap fixpoint (stage1 == stage2 == stage3 behavioral parity)
Soundness gate for self-hosting (jolt-d0r). stage1 = analyzer built by the Janet
bootstrap; stage2 = analyzer rebuilt by compiling jolt.ir + jolt.analyzer through
stage1 (self-host); stage3 = the same self-rebuild again. A corpus of programs
must produce identical results across all three.

Compared BEHAVIORALLY rather than by emitted code: emitted Janet forms embed live
setter/getter closures (identity varies per compile) and the IR carries
representation-level gensyms and pvec internals, so structural equality fights the
representation. Behavioral parity is the property that actually matters and is
representation-independent.

Also corrects the battery baseline 3920 -> 3919: 3920 was a lucky run; 3919 is the
stable value (the 9 timeout-prone tests can shift the count by one).
2026-06-06 18:27:14 -04:00
Yogthos
3638521e2d core: move juxt/every-pred/some-fn to Clojure
Three pure function combinators into the collection tier, as compositions of
mapv/every?/some/apply. Three more core-* primitives + table entries gone.
Battery 3916 -> 3920 (the canonical defs are more correct than the prior Janet
ones — some-fn returns the matching value, every-pred is properly boolean).
Conformance 218/218 all modes.
2026-06-06 18:09:42 -04:00
Yogthos
57b8ffcb9b core: move frequencies/group-by/not-empty/filterv/max-key/min-key to Clojure
New collection tier (core/20-coll.clj): six pure, eager fns expressed as
compositions of frozen primitives (reduce/assoc/get/conj/filter/vec). Six more
core-* primitives + table entries gone from the Janet kernel.

Two semantics to preserve, caught by the suite:
- max-key/min-key use the canonical multi-arity form (strict </> on the first
  pair, <=/>= in the fold) to reproduce the JVM IEEE-754 NaN behavior; a single
  reduce got the NaN cases wrong.
- frequencies/group-by base on (hash-map), not the {} literal: a struct map
  doesn't canonicalize collection keys across representations ({:a 1} literal vs
  (hash-map :a 1)), a PHM does — same reason the Janet impl used make-phm.

Conformance 218/218 all modes; battery holds 3916.
2026-06-06 18:02:34 -04:00
Yogthos
279c8e176b perf: generation-guarded inline cache for host-value protocol dispatch
Host-value protocol dispatch (a protocol extended to Number/String/… called on a
host value) recomputed the candidate type-tag list and walked the registry — up
to ~15 map lookups plus an array alloc — on every call. It now caches
(most-specific-host-tag, protocol, method) -> fn, guarded by a registry
generation that bumps in register-protocol-method; re-extending a protocol
invalidates the cache. The single-lookup fast paths (deftype protocol dispatch,
direct multimethod dispatch) are untouched — they're already cheap, so no cache
overhead there. The multimethod hierarchy-fallback cache (narrower: only
derive-based dispatch) is deferred.

Test in test/integration/dispatch-cache-test.janet covers correctness under
re-extension in all three modes. Conformance 218/218; battery 3916.
2026-06-06 17:38:44 -04:00
Yogthos
db08cecde8 perf: var-deref fast path; drop the memoized-getter indirection
var-get short-circuits to the root when no dynamic bindings are active (the
common case — the binding stack is only non-empty inside a `binding` block),
skipping the stack walk on every global deref.

The compiled indirect deref now emits (var-get (quote cell)) instead of routing
through a per-var memoized getter closure — one fewer call and no per-var closure
alloc. The cell is quoted so it embeds by reference: a bare table in argument
position is re-evaluated as a constructor, which deep-copies the cell (and any
atom in its root) on every call — that silently broke swap!/atoms/self-ref
lazy-seqs until the quote. Conformance 218/218 all three modes; battery 3916.
2026-06-06 17:31:16 -04:00
Yogthos
a3cc035782 compiler: direct-linking (call-site/unit, Clojure model) + :aot-core? flag
Calls compile direct (target value embedded) iff the compiling unit has
direct-linking on, the target isn't ^:redef/^:dynamic, and it's an
already-defined fn; otherwise indirect (live var deref, redefinable). Direct
emission is guarded to function values only — embedding a non-function would make
Janet evaluate it as code.

:aot-core? (init opt, default true; JOLT_AOT_CORE=0 disables) compiles the core
tiers with direct-linking on so core->core calls are direct. User/REPL code
compiles indirect by default, so redefining functions in the REPL works with no
annotation, exactly like Clojure. :aot-core? false makes core indirect too — the
whole language becomes redefinable. ^:redef / ^:dynamic on a target force it
indirect even inside a direct-linked unit.

Also fixes a pre-existing gap this surfaced: compiled def dropped ALL var
metadata (so ^:dynamic was silently lost under compilation, and ^:redef couldn't
work). def metadata now flows analyzer -> IR -> back end (new form-sym-meta host
fn, def-node :meta, var-setter-meta) and is applied to the cell.

Matrix test in test/integration/direct-linking-test.janet. Conformance 218/218
in all three modes under both :aot-core? true and false; battery holds 3916;
binary rebuilt.
2026-06-06 17:13:54 -04:00
Yogthos
a2805c4f51 vars: add per-var generation counter, bumped on root change
Foundation for the linking-mode work (jolt-5ed). Every var cell carries a :gen
that bumps on bind-root / var-set(root) / alter-var-root — i.e. on redefinition.
Behavior-neutral on its own; it's the substrate the upcoming work needs:
direct-linked (sealed) call sites can detect staleness, and redefinition-aware
dispatch caches can invalidate per-var (only a redefined var's callers miss,
not a global flush). Full suite green, conformance 218/218 in all three modes.
2026-06-06 16:47:44 -04:00
Yogthos
28ef15ea4b core: staged-bootstrap kernel tier; move second/peek/subvec/mapv/update to Clojure
First fractal turn of the multi-stage bootstrap (jolt-tzo). The Clojure part of
clojure.core is now loaded in ordered tiers under jolt-core/clojure/core/. The
kernel tier (00-kernel: second/peek/subvec/mapv/update) holds the structural fns
the self-hosted compiler itself uses; in compile mode it is bootstrap-compiled
into clojure.core BEFORE the analyzer is built, so the analyzer binds those names
to the Clojure definitions instead of a not-yet-defined forward ref. That removes
the circularity that previously forced these to stay in Janet — the five core-*
primitives and their init-core! entries are gone.

Mechanism: backend/bootstrap-load-source (generalized from compile-load) builds a
source string into a target ns via the bootstrap; rebuild-compiler! recompiles
the self-hosted compiler against the current core (the rail for future turns,
exercised by the new fixpoint test). api/load-core-overlay! walks the ordered
tiers, bootstrap-loading kernel tiers and self-hosting the rest.

Also fixes a latent evaluator bug surfaced by the move: a fn rebinds current-ns
to its defining ns while it runs and restores on normal return, but a fn that
THREW unwound past its own restore, leaking the ns. try now restores the
try-entry ns on the catch/finally path, so referred-symbol resolution survives a
caught error (this was breaking is/testing in the suite harness after any thrown
assertion). Net battery +3 (3913 -> 3916); conformance 218/218 in all three
modes; binary builds and runs the embedded tiers.
2026-06-06 13:15:17 -04:00
Yogthos
a6f0f8a6fe core: move last/butlast to the Clojure overlay
Continue the kernel shrink. last/butlast become canonical Clojure defs
(pure first/next/loop/recur) in the core.clj overlay, dropping two
realize-for-iteration uses from the Janet kernel.

second was the obvious next candidate but can't move: the self-hosted
compiler front-end (analyzer.clj) calls it, and the compiler has to
compile the overlay, so anything it uses must already exist as a Janet
primitive. Documented that as a third clause of the safe-to-move rule.

Suite green, clojure-test-suite battery holds 3913 pass / 58 clean,
binary builds and runs the embedded overlay.
2026-06-06 12:08:00 -04:00
Yogthos
c8810ee4aa core: move nfirst to the Clojure overlay
Same leaf-fn migration as ffirst/fnext/nnext. Full suite green, battery 3913.
2026-06-06 11:49:09 -04:00
Yogthos
30a308f3ef core: start moving clojure.core to Clojure (overlay loaded at init)
Phase 4 kernel-shrink seam. jolt-core/clojure/core.clj holds the Clojure portion
of clojure.core; api/init loads it into the clojure.core namespace after
init-core! interns the Janet primitives, routed through eval-toplevel so it
compiles via the self-hosted pipeline (or interprets when :compile? is off).

First three fns moved off the Janet side: ffirst, fnext, nnext (leaf, no internal
callers). Same results compiled or interpreted; full suite green; battery holds
3913.
2026-06-06 11:45:31 -04:00
Yogthos
984af02532 loader: drop write-only compiled-form cache and its unused helpers
compiled?/get-compiled-forms/clear-compiled-cache were never called, and
load-ns only ever wrote to :compiled-cache with no reader. All dead once the
self-host path landed. Removed the helpers, the cache writes, and the
:compiled-cache env field.
2026-06-06 11:40:27 -04:00
Yogthos
497970029d self-host: make it the only compile path; drop the bootstrap as a selectable path
eval-toplevel now routes every compiled form through backend/compile-and-eval
(analyzer -> IR -> Janet back end). The bootstrap compiler (compile-ast) stays
only to bootstrap jolt.ir/jolt.analyzer in backend/compile-load. Dropped the
:selfhost? flag and JOLT_SELFHOST env plumbing — self-host is the default.

Three correctness fixes the full-suite-under-self-host sweep exposed:
- analyzer/back end: recur into a variadic arity now compiles. The arity fn
  gets an ordinary positional rest param (the collected seq), with a dispatch
  wrapper, so (recur fixed... rest-seq) is a clean self-call like Clojure --
  instead of punting to the interpreter, which hangs on this (jolt-4df).
- host_iface: interop heads (.foo/.-field/Foo.) and ns-management forms
  (all-ns, create-ns, resolve, ...) now mark uncompilable so they fall back
  instead of compiling to a bad call.
- back end: named-fn self-reference ((fn self [n] ... (self ...))) binds the
  fn name via a var.

Full unit+integration suite green; clojure-test-suite battery holds 3913.
2026-06-06 11:34:02 -04:00
Yogthos
34b880f0d6 conformance: guard the self-hosted pipeline too (218/218 in all three modes)
conformance-test now runs every case under interpret, the bootstrap compiler, AND
the self-hosted pipeline (portable Clojure analyzer -> IR -> Janet back end). All
three pass 218/218, so the self-hosted compiler can't silently regress in CI.
2026-06-06 10:29:36 -04:00
Yogthos
3afd1c6142 self-host: analyzer compiles to fast bytecode (fib(30) 0.52s)
The self-hosted compiler now genuinely compiles to native Janet bytecode rather
than falling back to the interpreter:
- bootstrap fixes that let it compile the analyzer: (def x) with no init, and an
  empty (do) (declare expands to one) no longer error;
- analyzer reordered so only the mutually-recursive  is forward declared
  (a deeply-nested forward ref to analyze-fn/analyze-try miscompiled to nil);
- back end emits native Janet ops for hot primitives (+,-,*,<,>,<=,>=,inc,dec)
  on clojure.core refs, like the bootstrap's core-renames.

Result: self-hosted fib(30) runs in 0.52s (was ~50s interpreted). Full
clojure-test-suite via JOLT_SELFHOST runs compiled at 3869 pass (some compiled-
vs-interpreted divergences remain to chase toward full parity — jolt-4xc).
Default suite green; conformance 218/218; jank-conformance up to 135.
2026-06-06 10:25:05 -04:00
Yogthos
06d5f51d4a self-host: compile the analyzer (full-suite parity, fast); fix interp multi-arity
The portable analyzer now refers the host contract + IR ctors unqualified (host
form predicates renamed form-* to dodge core-renames), so the bootstrap compiles
it via its plain :var path — no qualified-ref compilation needed. ensure-analyzer
compile-loads jolt.ir + jolt.analyzer as native bytecode.

Result: the self-hosted pipeline (portable Clojure analyzer -> IR -> Janet back
end) runs the FULL clojure-test-suite at 3913 pass — parity with the interpreter
baseline — and fast (no timeouts), via JOLT_SELFHOST=1. Conformance 218/218.

Also fixes a real interpreter bug: multi-arity dispatch stored the variadic clause
by fixed-count (colliding) and only matched exact counts, so (f a b c..) on an
[a b & more] clause threw. Now the variadic clause dispatches for any count >= its
fixed arity.

Remaining (jolt-4xc): the compiled analyzer still errors analyzing a multi-arity
fn literal (falls back to the interpreter, now correct); compiling that is the
last coverage gap before flipping the self-hosted pipeline on by default.
2026-06-06 09:59:28 -04:00
Yogthos
98bf389f82 compiler: resolve :as aliases in macro detection; fix analyzer compile-ns
- compiler/resolve-macro now resolves :as aliases (like resolve-var), so aliased
  macro calls (t/is) are recognized and compiled via expansion instead of falling
  back. Compile-mode suite holds at 3932.
- host/current-ns reads a backend-stashed :compile-ns rather than ctx-current-ns:
  the interpreter rebinds current-ns to a fn's defining ns while it runs, so the
  interpreted analyzer (defined in jolt.analyzer) would otherwise mis-resolve the
  user's compile ns. Fixes def/intern targeting the wrong namespace.

Analyzer stays interpreted (compiling it needs qualified-ref compilation, which
regresses the clojure.test shim — jolt-4xc). Self-host pipeline correct (218/218),
full suite green.
2026-06-06 09:48:52 -04:00
Yogthos
b3a80507cc destructuring via shared expander; fix sequential let + nth nil-default
core-let now expands destructuring (Clojure's destructure algorithm: nth/nthnext/
get over gensym roots) so both the interpreter and the compiler see only plain
bindings — the compiler compiles destructuring for free, and it's a step toward
moving destructuring out of the kernel. Handles vector (& rest, :as, nested), map
(:keys/:strs/:syms incl. namespaced, :as, :or, explicit pairs).

Two real bugs fixed in passing:
- compiler let*/loop* analyzed all binding inits in the OUTER scope, so a later
  binding couldn't see an earlier one ((let [x 1 y x] y) miscompiled). Now scope
  accumulates across bindings.
- core-nth treated an explicit nil not-found as 'no default' and threw on out of
  bounds; (nth coll i nil) now returns nil (Clojure semantics), via arity.

Baselines held: conformance 218/218 both modes, clojure-test-suite 3913 interp /
3932 compile, destructuring-spec green.
2026-06-06 09:36:36 -04:00
Yogthos
b6e41bf499 self-host: JOLT_SELFHOST suite mode; keep analyzer interpreted for now
suite-worker honors JOLT_SELFHOST=1 to route the clojure-test-suite through the
self-hosted pipeline (for validating once the analyzer compiles).

The analyzer stays interpreted: compiling it via the bootstrap miscompiles
(qualified refs regressed compile mode 3932->1181; compile-loading yields
corrupted IR) — tracked in jolt-4xc. Correctness is unaffected (the self-hosted
pipeline passes 218/218 conformance); only speed is pending.
2026-06-06 06:41:36 -04:00
Yogthos
f9df794475 self-host: compile loop/recur, recur-in-fn, and try
Analyzer threads an env (locals + recur target) instead of a bare locals set, and
compiles loop*/recur, recur directly in a fixed-arity fn (each arity's name is its
recur target -> self-call), and try/catch/finally (-> Janet try + defer). recur
into a variadic arity isn't a recur target, so it falls back to the interpreter
rather than miscompiling the rest seq. Still 218/218 conformance via the pipeline,
now with these forms compiling natively instead of interpreting.
2026-06-06 06:26:22 -04:00