Loading these libs via require worked (load-ns-source interprets, macros
expand lazily) but the same code inlined by uberscript routes through
eval-toplevel and compiled, surfacing four gaps:
- a ^{:map} metadata def name reads as (def (with-meta name m) v); the
analyzer died extracting the name (config.core's defonce env). It now
throws uncompilable so the interpreter, which handles it, takes over.
- declare was a no-op, so a compiled forward reference to a declared
name that collides with a janet root binding bound to the host fn
(selmer.parser's (declare parse) compiled to janet's 1-arg parse).
declare now expands to no-init defs, the interpreter interns them,
and the analyzer routes no-init def to the interpreter.
- class? was missing (selmer.util's exception macro calls it at
expansion time). Always false, like ratio? — no Class objects here.
- require of an unlocatable namespace silently left an empty ns behind,
deferring the failure to an unresolved symbol far from the cause. It
now throws like Clojure's FileNotFoundException. Namespaces entered
in-session count as loaded (Clojure puts them in *loaded-libs*), and
the SCI bootstrap opts out via :lenient-require? since its
clj-targeted requires can't all exist on this host.
recompile-defns! is the defn analog of recompile-macros!: pre/at-kernel
overlay defns (00-syntax's destructure and friends; the kernel tier too in
interpret mode) load as interpreted closures, the evaluator stashes their fn
source on the var (:defn-src, scoped by a flag only api/load-core-overlay!
sets), and the end-of-init pass compiles them and swaps the var root. With
that in place, keys/vals/empty? — the fns the 00-syntax expanders call at
expansion time — move to the top of 00-syntax as raw fn* defs (canonical:
keys/vals project (seq m), so sorted maps come back in comparator order and
(keys {}) is nil; empty? keeps O(1) count dispatch with seq's cell check only
for the lazy/list fallback). The sorted tier drops its now-dead :keys/:vals
ops.
Correctness fixes that surfaced once the gate was run with a REAL exit code
(the previous 'jpm test | grep' gates reported grep's exit and masked spec
failures across #48-#50):
- map conj is strict again: a non-nil/non-map arg must be a 2-element vector
('Vector arg to map conj must be a pair'), and merge inherits it — the
batch-2 canonical merge had silently dropped the validation
- conj onto a lazy seq prepends (it fell into the MAP fallback); upstream
clojure.data/diff relies on (conj seq x) via set/union over keys, so diff
now matches Clojure exactly
- (seq {}) / (seq #{}) / empty phm are nil, not ()
- key/val are strict (a plain vector is not an entry); find mints a REAL
entry as the first entry of a one-entry map, nil values intact
- the sci avoid-method-too-large stub passes its registry map through
instead of returning a raw host table (strict conj rejected it; sci's
clojure-core registry is also no longer discarded)
Test updates: lazy-infinite pins take-nth realization at 5 (was 7 — the
canonical lazy impl realizes fewer); self-host asserts the analyzer IS loaded
in interpret mode (compiled expanders, PR #50) and is NOT in the
:compile-macros? false oracle. 18 new maps-spec rows.
Gate: jpm test exit 0 (verified directly, not through a pipe), conformance
326x3, suite 4698 >= 4660.
* test: adapt jank's form/reader tests into spec suites; fix case no-match
Vendoring jank's behavior (not the project): we base our own copies on the
jank test corpus to close coverage gaps, but maintain them ourselves since
jank may diverge. Two new spec batteries (jank-isms translated to Jolt:
letfn* -> letfn, jank catch types -> :default; platform-specific bigdec/
biginteger/ratio/uuid/##Inf/unicode cases omitted):
- test/spec/forms-spec.janet (52): case, fn (arity/variadic/closure/recur/
named), let, letfn, loop, try, if/do/def/call — incl :throws regression
cases (no-match, bad params, nil call).
- test/spec/reader-forms-spec.janet (22): #() (% %N %&), #' var-quote,
^metadata, syntax-quote (gensym/unquote/splice).
Fix surfaced by adaptation: case with no matching clause and no default now
throws 'No matching clause' (Clojure semantics) instead of returning nil
(00-syntax). Gate green incl full jpm build+test.
Other gaps the adaptation surfaced are filed (tests adjusted to jolt's
current behavior + a comment, not silently dropped):
jolt-vdo case duplicate test constants not rejected
jolt-w2v loop bindings not sequential (later init can't see earlier)
jolt-6x1 #() %& miscomputes arity with a higher positional (%2 …)
jolt-xl0 ^meta not attached to collection literals ({}/[]/#{})
jolt-265 syntax-quote doesn't fully-qualify core syms to clojure.core/
jolt-edb syntax-quote ~/~@ not processed inside set literals
* core: fix 5 Clojure-conformance gaps surfaced by jank tests
All from adapting jank's form/reader tests; each fix verified in interpret +
compile modes and the spec tests now assert the correct behavior.
- jolt-vdo: case now rejects duplicate test constants at expansion (Clojure
compile error), via bootstrap-safe duplicate detection (00-syntax; analyzer.clj
uses case during its own build, so seed-only fns).
- jolt-w2v: loop bindings are now sequential like let — a later init can
reference an earlier binding. Fixed the interpreter loop* (accumulating scope)
and the back end emit-loop (bind initial inits in a sequential Janet let before
entering the recur target).
- jolt-6x1: #() reader computes the fixed arity from the MAX positional (%2 ->
[p1 p2 & rest]); % and %1 unify; unused lower slots get placeholder params.
- jolt-xl0: ^meta on collection literals ({}/[]/#{}) now attaches — read-meta
passes the NORMALIZED metadata map to with-meta (was the raw meta-form).
- jolt-edb: syntax-quote processes ~/~@ inside set literals (new __sqset builder
in core + set branches in syntax-quote* and syntax-quote-lower).
Deferred: jolt-265 (fully-qualify core syms to clojure.core/ in syntax-quote) —
it passes conformance but breaks the standalone uberscript (the ns macro emits
unqualified require/in-ns, which then qualify and break bundled require/alias).
Reverted to bare (functionally resolves); re-opened with the finding.
Gate green incl full jpm build + jpm test: conformance 269x3, suite >=4034/67,
fixpoint, self-host, sci 422/0, uberscript, all unit + spec (forms 55, reader 31).
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* core: Stage 2 Task 2 tier 4b — compile ns (+ use/import/refer-clojure)
ns becomes a macro (00-syntax) expanding to (do (in-ns 'name) (require …)
(use …) (import …) (refer-clojure …)) — matching Clojure, where require is
a fn and ns expands to require calls. use/import/refer-clojure join the
namespace ops as ctx-capturing clojure.core fns (use-impl/import-impl/
refer-clojure-impl, interned by install-stateful-fns!). Removed the ns
interpreter special arm; dropped ns/use/import from host_iface special-names
and loader stateful-head?.
Placement matters: ns lives in the FIRST overlay tier (00-syntax), because
the self-hosted analyzer build is triggered while 10-seq loads and processes
jolt.analyzer's own (ns …) form — ns must exist by then. Its body resolves
fn/map/reduce/cond at expansion time, by which point all of 00-syntax has
loaded. The bootstrap compiler (compiler.janet) still PUNTS ns/in-ns/require/
use/import (it compiles analyzer.clj/ir.clj, whose ns forms must fall back to
the interpreter that expands the macro) — only the self-hosted analyzer
compiles them.
use-impl fixes a latent bug: it now refers the used ns's vars into the
CURRENT ns (the old special interned a ns into itself, a no-op).
ns/use/import now compile + interpret as plain invokes. fallback-zero: ns off
must-punt, onto must-compile.
Gate green: conformance 267x3, fallback-zero 38/4, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, sci-bootstrap, clojure-test-suite
>=4034/67, namespace, deps-loader, cli, aot, embedded-stdlib, uberscript,
features 78/78, all unit + spec.
* test/core: cover ns form + :use; fix use-impl spec parsing
Checking test coverage for the namespace tiers surfaced a real bug nothing
exercised: use-impl checked (array? s) to find a spec's ns symbol, but a
vector spec like [src.x] is a pvec/tuple, not a Janet array — so (:use ...)
and standalone (use ...) errored. Fixed to coerce (pvec->array) then take the
head when the spec is indexed (matching require-impl).
Added coverage that was missing:
- conformance (x3 modes): the (ns …) form itself + :use refers.
- namespaces-spec: (ns …) form, :use, standalone use.
The :use/(use …) path had ZERO tests before, which is why the bug slipped
through earlier tiers.
EBNF needs no change — it's a reader/syntax grammar with no special-forms
enumeration; ns/require/protocol syntax is unchanged (the destructuring note
was already updated in jolt-f79).
Gate green: conformance 269x3, fallback-zero 38/4, self-host, sci-bootstrap,
bootstrap-fixpoint, staged-bootstrap, clojure-test-suite >=4034/67, namespace,
all unit + spec (namespaces 27/27).
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* core: compile macro expanders via staged bootstrap (Stage 2 Task 1)
Macros are now compiled, not interpreted, by steady state — matching
Clojure (macros are ordinary compiled fns the Java seed compiles for
clojure.core) and ClojureScript (macros compiled, invoked at compile
time). Neither reference keeps an interpreted-closure fallback.
The early macros (00-syntax) are defined while the self-hosted analyzer
is still being bootstrapped (it builds lazily after the overlay loads),
so macro-compile-hook returns nil and they get an interpreted closure.
The bootstrap compiler.janet can't substitute (it punts on syntax-quote,
which nearly every expander uses).
Fix = staged bootstrap, the same pattern as the compiler fixpoint:
defmacro stashes the expander source on the var (:macro-src) plus a
:macro-uses-env flag; once the overlay + analyzer are fully built,
backend/recompile-macros! (via ensure-macros-compiled! at the end of
load-core-overlay!) compiles each stashed expander through the now-live
analyzer and rebinds the var, marking :macro-compiled. Idempotent;
&env/&form macros keep the interpreted closure (the compiled fn* has no
such params). The interpreter is now a build-time crutch, gone by
steady state.
Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring
(which the analyzer punts on) to a plain rest param + (first rest), so
all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79.
47/47 overlay macros compiled, 0 interpreted; user macros compile
immediately post-init. Gate green: conformance 267x3, fallback-zero
31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap,
lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap,
features 78/78, all unit+spec, core-bench neutral.
* core: wrap macro expanders in fn (not fn*) so destructured arglists compile
Follow-up to the staged macro-compile change. The macro-recompile pass
wrapped each expander in the raw fn* primitive, which punts on a
destructuring rest param — so if-not/if-let/if-some/assert (using the
canonical '& [else]') couldn't compile and had been rewritten to plain
rest params as a workaround.
Root cause: fn* is the primitive; the fn MACRO is what desugars
destructuring (rest, map, nested) into the body before lowering. Wrapping
expanders in fn instead of fn* compiles any destructured macro arglist
uniformly, so the workaround is unnecessary — reverted those 4 macros to
the canonical '& [else]' forms.
Net: rest-destructuring is fully compiled for all normal code (fn/defn/
let/macro params). Only the hand-written raw fn* primitive still punts
(jolt-f79, downgraded to P4 — falls back to interpreter, still correct).
47/47 overlay macros compiled. Gate green: conformance 267x3,
fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host,
staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67,
sci-bootstrap, features, all unit+spec.
* core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79)
jolt-f79 asked to compile destructuring in fn* rest params. Checking
against Clojure inverts the premise: Clojure's fn* REJECTS destructuring
at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding
form, expected symbol'). So the self-hosted analyzer was already correct
— fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn
MACROS desugar destructuring. The real defect was the interpreter
leniently destructuring raw fn*, and defn emitting raw fn* to rely on it.
Changes:
- evaluator: fn*/let*/loop* now reject non-symbol binding forms with
Clojure's exact messages (require-symbol-params/plain-sym?), so the
interpreter agrees with the analyzer + Clojure.
- 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring
params desugar; unnamed, so self-recursion still resolves via the var.
- 00-syntax: completing that exposed a real gap — the overlay destructure
fn didn't handle kwargs (a map pattern bound against a fn's sequential
rest); it had only worked via the interpreter's destructure-bind. Added
the seq->map coercion to the map? branch (sequential: 1 map elt => that
map, else apply hash-map), matching destructure-bind so interpret ==
compile.
Net: fn*/let*/loop* are plain-symbol primitives across interpreter,
analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro
params, incl kwargs & {:keys}) compiles through the macros with no
interpreter fallback. Regression spec added.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44,
clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all
unit+spec (destructuring 50/50).
* docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79)
grammar.ebnf: the destructuring note already attributed patterns to the
binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES
they desugar to take plain symbols only (a non-symbol binding errors, as
in Clojure).
self-hosting-compiler.md: the 'compile destructuring via a shared
destructure expander instead of falling back' item is done (and its
jolt-7dl ref was stale) — destructuring now compiles through the
fn/let/loop/defn macros' desugaring; the primitives reject patterns.
* core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy
First slice of moving stateful forms onto the compile path (jolt-eaa).
- loader stateful-head?: drop syntax-quote (the analyzer's `handled` set
already compiles it; routing it to the interpreter was redundant).
- host_iface special-names: drop definterface/extend/proxy (stub macros
expanding to def/nil — their expansions compile once unpunted).
- letfn stays interpreted: its let* expansion needs letrec semantics
(mutual recursion between the fns), which sequential compiled let* lacks
— a later tier.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + protocol/multimethod/macro specs.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
adds self-hosted compiler is functionally:
- The default compile path is the portable pipeline using jolt.analyzer (Clojure) → host-neutral IR → backend.janet.
- The analyzer is itself Clojure, compiled by jolt for true self-hosting.
- bootstrap-fixpoint passes (stage1 == stage2 == stage3): rebuilding the compiler on its own output.
- clojure.core is now self-hosted in the overlay.
- Stateful forms (defmacro/ns/deftype/defmulti/require/in-ns) are interpreted by design.
Yogthos
Dmitri Sotnikov