Makes the inc8 fixpoint the actual build. host/chez/bootstrap.ss loads a seed
(prelude, image) pair and rebuilds the clojure.core prelude + compiler image from
source via the on-Chez compiler — read/analyze/emit all on Chez, zero Janet.
The seed (host/chez/seed/{prelude,image}.ss) is the checked-in bootstrap
compiler, minted once via the fixpoint (driver/mint-chez-seed* iterates
bootstrap.ss from the Janet-emitted pair to a joint byte-fixpoint). It's a joint
fixpoint: rebuilding from an up-to-date seed reproduces it exactly. So a fresh
checkout + Chez (no Janet) yields a working jolt.
test/chez/bootstrap-test.janet spawns only chez, asserts the rebuilt artifacts
match the seed byte-for-byte and compile+run real cases. Drift (seed sources
changed) fails the test with a re-mint pointer; host/chez/seed/README documents
re-minting.
Extends the fixpoint beyond the compiler image to the whole emitted system.
emit-image.ss now handles macros (defmacro -> bare expander fn + def-var! +
mark-macro!) and re-emits the clojure.core prelude (all tiers + stdlib) on Chez
via jolt-emit-prelude; driver's emit-image-on-chez takes an emit-fn arg.
The prelude converges at pstage3==pstage4 (one stage later than the compiler's
stage2==stage3) because macro expanders bake an auto-gensym id at emit time, so a
Janet-emitted macro carries a different id than a Chez-emitted one — only once
both stages load a Chez-emitted prelude does it stabilize.
fixpoint-test now proves: compiler stage2==stage3, prelude pstage3==pstage4, and
the fully Chez-emitted system (Chez prelude + Chez image, no Janet artifact in the
loop) compiles+runs real cases. 10/10.
The zero-Janet spine proves the on-Chez analyzer/emitter compile arbitrary
Clojure faithfully. This proves the stronger property: the on-Chez compiler
reproduces itself. emit-image.ss re-emits the compiler sources (jolt.ir +
jolt.analyzer + jolt.backend-scheme) ON CHEZ via the loaded image; feeding it
stage1 (the Janet cross-compile) yields stage2, feeding stage2 yields stage3.
stage2 and stage3 are byte-for-byte identical, and stage2 is a working compiler
(real cases compile+run through it). stage1 differs from stage2 only in gensym
numbering, so the fixpoint is stage2==stage3.
driver: emit-image-on-chez / program-emit-image spawn a fresh chez per stage
(clean gensym state). test/chez/fixpoint-test.janet gates it (skips without chez).
(require '[clojure.string :refer [blank?]]) then an unqualified blank? now
resolves. chez-register-spec! registers :refer names (in addition to :as) into a
refer table; hc-resolve-cell's unqualified branch consults it before clojure.core.
Zero-Janet corpus parity 2293 -> 2295 = the Janet-hosted oracle's exact pass
count. The self-hosted Chez compiler (read -> analyze -> emit -> eval, no Janet)
now compiles every corpus case the Janet-hosted compiler does, with 0
divergences. Remaining failures are shared runtime breadth (host interop,
futures, runtime eval) deferred to Phase 4 / jolt-r8ku. Floor 2295.
The batched zero-Janet runner wrote one case per TSV line, so a multi-line
source (e.g. a ;comment\n inside a map literal) split the line and the case was
truncated -> a false "apply non-procedure" crash. Escape \n/\t/\\ when writing
the cases file and unescape in the runner before eval.
Zero-Janet corpus parity 2288 -> 2293 (the 5 comment-in-map cases), 0
divergences — now within 2 of the Janet-hosted oracle (2295). Floor 2293.
(require '[clojure.string :as s]) then s/foo crashed "Unknown class s": the
Chez analyzer resolved a qualified symbol's ns literally, and there was no
alias table (ns.ss jolt-ns-aliases was a stub, require a no-op). Add an alias
table + chez-register-spec! (parses [ns :as a]) in ns.ss; hc-resolve-cell now
resolves a qualified ns through it; compile-eval pre-registers a form's
require/use :as aliases before analysis (analysis precedes the runtime require,
so the runtime require staying a no-op is fine). The batched gate runner clears
the alias table between cases.
Zero-Janet corpus parity 2159 -> 2240, 0 divergences. spine-test 35/35.
Point the Chez-HOSTED analyzer at the full parity corpus (read -> analyze ->
emit -> eval, all on Chez, no Janet) and close the divergences so the
self-hosted compiler is faithful: 0 divergences, 2159/2494 pass.
Keystone: the on-Chez emitter ran with prelude-mode off, so every call to a
non-native clojure.core fn tripped the "unsupported stdlib fn" out-of-subset
guard. The zero-Janet spine always has the full prelude loaded, so turn
prelude mode on in compile-eval.ss (22% -> 84% pass on a sample).
Faithfulness fixes (each was the Chez host/reader diverging from the Janet
analyzer; fixed in the keeper, not the seed):
- emit-const read a char's codepoint via (get v :ch) — the Janet rep; on Chez a
char is native. Route through a new form-char-code host-contract fn (41 cases).
- next over a lazy seq returned the empty-list terminator (truthy), not nil, so
butlast and other (if (next s) ...) loops ran one step too far — broke
some->/some->>/cond->>.
- reader: radix literals (2r1010/16rFF/36rZ), #^ deprecated metadata, ^meta on
collections (lowers to a runtime with-meta form like the Janet reader),
map-literal source order (values eval left-to-right), and nested syntax-quote
over a literal collapses at read time.
- keyword "a/b" splits into ns/name like the seed (destructure {:keys [x/y]}).
- form-syntax-quote-lower implemented on Chez (was a throwing stub).
7 divergences allowlisted: the same print-method-multimethod / host-class set
the prelude gate defers. 328 crashes remain = shared runtime breadth (host
interop, missing core fns, eval/load-string) deferred to Phase 4 / jolt-r8ku,
not compiler faithfulness.
Gate + speedup: test/chez/run-corpus-zero-janet.janet (floor 2159). Its batched
runner (driver/eval-corpus-zero-janet) runs every case in ONE chez process —
load the runtime once, guard + reset the user namespace per case — instead of a
fresh process per case: 1379s -> 1.6s.
spine-test 35/35; Janet gate 151/0; prelude parity 2295/2494 unchanged, 0 new
divergences.
The on-Chez analyzer (inc6a) skipped macros, so let/when/->/defn didn't
expand from source. Each core/stdlib defmacro now emits into the prelude as
(def-var! ns name <expander fn>) + (mark-macro! ns name); form-macro?/
form-expand-1 on Chez look up the macro flag (rt.ss var-macro-table) and
apply the expander to the unevaluated arg forms, and the analyzer re-analyzes
the result. The expander's syntax-quote template was lowered to construction
code at cross-compile time, so it builds the expansion via __sqcat/__sqvec/
__sqmap/__sqset/__sq1 (new host/chez/syntax-quote.ss) as Chez reader forms.
Emit the bare (fn ...), not (def NAME (fn ...)): analyzing a def would
host-intern! NAME as a non-macro stub in the build ctx, and that stub makes a
later (require '[stdlib-ns]) skip loading the real macro — with-pprint-dispatch
then resolved as a fn and returned its unexpanded template. Wrapping the
lambda in def-var! manually never interns NAME. Fuller build-ctx isolation
(so stdlib cases pass instead of crash) tracked in jolt-lpvi.
__sqset builds a real set VALUE, not the reader's tagged-set form — a runtime
`#{~@xs} must be a set, not a map. form-set? additionally recognizes a pset so
a macro template's #{...} expansion still re-analyzes as a set literal.
spine-test 35/35 (20 macro cases: when/when-not/let/->/->>/and/or/cond/if-not/
defn run zero-Janet from source). Prelude parity 2280->2295, 0 new divergences.
Full Janet gate green.
Cross-compile jolt.ir + jolt.analyzer + jolt.backend-scheme to Scheme def-var!
forms via the existing Janet emit pipeline (driver/emit-compiler-image) and run
them ON CHEZ over a Scheme jolt.host impl. A macro-free Clojure expression now
compiles and runs with no Janet in the loop: read (reader.ss) -> analyze
(jolt.analyzer on Chez) -> IR -> emit (jolt.backend-scheme on Chez) -> eval.
host/chez/host-contract.ss is the jolt.host contract on Chez (the portable seam
the cross-compiled analyzer/emitter call): form-* over the Chez reader's forms,
resolve-global/compile-ns/host-intern!/late-bind? over the var-cell registry. ctx
is an opaque record carrying the compile ns. Native-op names are declare-var!'d
into clojure.core so +, *, <, ... classify as :var and the emitter's native-op
path lowers them. form-macro? is a #f stub and macro expansion / syntax-quote /
record hints are stubs for inc6b (runtime macros, jolt-r8ku).
host/chez/compile-eval.ss is the spine entry (read-string -> analyze -> emit ->
eval). driver gains emit-compiler-image / ensure-compiler-image (image caching)
and program-zero-janet / eval-zero-janet.
Two bugs fixed in the keeper, not reproduced:
- the Chez reader stores an unqualified symbol's ns as #f, but the analyzer tests
(nil? ns); hc-sym-ns normalizes #f/'() -> jolt-nil. Without it every handled
special (if/do/fn*) misanalyzed as a plain invoke.
- char (int->char) was missing from clojure.core on Chez; the emitter's
chez-str-lit needs it for keyword/string consts. Added jolt-char to converters.ss.
Gate: test/chez/spine-test.janet 15/15 (Chez-hosted analyzer value == Janet-hosted
oracle through the same emitter/RT; only the analysis host differs). Full Janet
gate green (150 files). driver.janet is in the prelude fingerprint so the cache key
moved; prelude content is unchanged. jolt-chez fingerprint/ensure-prelude made
public for the test harness.
driver.janet now compiles IR via the portable Clojure emitter (jolt.backend-
scheme) instead of emit.janet, at every entry point (compile-program, emit-core-
prelude, eval-e-with-prelude). The emitter is loaded into the ctx and called like
the analyzer. emit.janet stays only as the emit/program string-wrapper until
program assembly ports to Clojure with compile-from-source; its emit fn is no
longer called anywhere (emit-test's truthy-elision helper now uses the new
d/scheme-emit too). This takes the IR->Scheme emitter off Janet.
A form-by-form diff of the two emitters over the whole prelude found one gap:
emit-const missed char literals because a :jolt/type-tagged struct is not a plain
jolt map? — switched to the form-char? host contract. Diff then 0.
jolt-chez prelude fingerprint now includes backend_scheme.clj + host_iface.janet.
Gate: full prelude corpus 2280/2494, NEW divergence 0, same buckets as the Phase-2
emit.janet floor (36 emit-fail, 170 crash) — the Clojure emitter is byte-for-
behavior identical. emit-test 331/331 (now via the Clojure emitter), emit-parity
58/58. jolt-duot.
Close the remaining Phase-2 stdlib parity gaps.
clojure.set: pure Clojure over core, so just added to the prelude stdlib tier
(driver.janet stdlib-ns-files + jolt-chez fingerprint), same as clojure.edn.
clojure.math: not a .clj on the seed (native math/ bindings via jolt-h79), so
Chez gets its own host/chez/math.ss def-var!ing each fn over Chez native flonum
math. The analyzer already knows the ns (api.janet install-clojure-math!), so
refs lower to var-deref. Also adds the missing 'long' coercion to converters.ss
(int's sibling; several math cases wrap in long).
clojure.pprint: dropped the 2-arity's (binding [*out* writer] ...). *out* isn't
a bindable var in jolt — printing routes through the host (dyn :out) seam, so
the binding never redirected anything; it only made the defn uncompilable, which
the seed tolerated via interpreter fallback. Chez has no fallback, so the whole
pprint defn died. Dropping it is behavior-preserving (writer was always ignored)
and lets pprint compile cleanly. Both corpus cases pass.
Corpus parity 2259 -> 2280, crashes 191 -> 170, 0 new divergences. Floor raised.
New unit test test/chez/_stdlib.janet (27/27). Full Janet gate green (147 files).
Chez-side recursive-descent Clojure reader (host/chez/reader.ss) producing the
same jolt forms the Janet reader yields, behind the read-string / __parse-next /
__read-tagged seams the seed registers in eval_runtime.janet. That lights up the
whole *in* read family — read, read+string, with-in-str (read) — plus read-string
and read-string metadata, none of which needed an analyzer change (read-string is
a clojure.core seam, jolt-nil on the prelude until now).
Reader output is pinned to the Janet reader's shapes: numbers coerce to flonum
(the all-double model emit-const uses, else a read int isn't = a source int),
sets read as the {:jolt/type :jolt/set :value [...]} FORM, #tag/#inst/#uuid/#regex
as tagged forms (no data reader applied — read-string never evaluates), ^meta on
a symbol, and the ' ` ~ ~@ @ reader macros. clojure.edn is added to the prelude
tier; its edn->value builds the real set/tagged values and __read-tagged reuses
the inc X #inst/#uuid constructors. The reader-arity edn/read stays a lazy gap
(drain-reader is Janet-coupled) — read-string is the live path.
eval / load-string / runtime defmacro are still out: they need the compiler at
runtime, which is Phase 3 (self-host). Chez-only change, no Janet gate.
Parity 2238 -> 2259, 0 new divergences. _reader 47/47; all chez unit tests green;
emit-test 331/331.
list? was nil on Chez because one cseq record backs both lists and lazy/
realized seqs. Add a list? marker field (cseq v2) set only on the HEAD cell
of a list -- (list ...), quoted list literals, cons, reverse, conj onto a
list. rest/next/seq/map therefore yield unmarked seq cells, so they are
seqs and not list?, matching the seed (where rest-of-a-list is a non-list
seq). Empty () is treated as a list.
vector?: drop the map-entry exclusion. Clojure's MapEntry implements
IPersistentVector and the seed agrees -- (vector? (first {:a 1})) is true.
Only dot-forms' coll dispatch read jolt-vector?, where a 2-vector entry is
correct.
clojure.walk + clojure.template join the prelude stdlib tier. The driver
now evals each stdlib ns's requires -- and the ns form's (:require ...)
clause -- so an aliased ref (template's walk/postwalk-replace) resolves at
emit time instead of lowering to an Unknown class host-static. ns forms are
evaled for that side effect but not emitted, so the runtime *ns* doesn't
leak to the last stdlib ns.
Parity 2163 -> 2176, 0 new divergences. New test/chez/_walk.janet 39/39.
Bring the clojure.string namespace up on Chez so aliased refs like s/split,
s/upper-case, s/join, s/replace resolve and run.
Three pieces. (1) The Chez AOT driver analyzes the whole user form before any
require runs, so a (require '[clojure.string :as s]) never registered the
alias in time; eval-e-with-prelude now recursively pre-evals require/use forms
against the ctx, which loads the aliased ns and registers the alias so the
analyzer resolves s/X to a clojure.string var. (2) emit-core-prelude emits
stdlib namespaces (clojure.string) as their own def-var! tier through the same
analyze->emit pipeline, so the runtime var-deref resolves. (3) natives-str.ss
def-var!s the str-* primitives clojure.string.clj is written over (upper/lower/
trim/triml/trimr/find/reverse-b/join/split/replace/replace-all), plus no-op
require/use. Regex split keeps interior empties and honors the limit (ported
the seed re-split); regex replace does $N backref expansion and fn replacement
(ported replacement-for). new RT/clj files added to the prelude fingerprint.
Corpus prelude floor 2026 -> 2078 (+52), 0 new divergences. _strns 28/28 vs
build/jolt. Four previously-CRASHING cases now emit+run and surface pre-existing
gaps (read-line vector eval-order x3, instance? clojure.lang.Atom) — allowlisted
with notes. full jpm test + conformance x3 green.
sequence and transduce were seed natives nil on the prelude; the stateful
transducer arities (take-nth/map-indexed/partition-by, overlay) drive a
volatile via volatile!/vswap!/vreset!/deref, also unshimmed — so the whole
(sequence xform coll) / (transduce xform f coll) surface crashed.
natives-xform.ss: native volatiles (a jvol record + volatile!/vreset!/
vswap!/volatile? + a deref arm); transduce/sequence built on the existing
into-xform / reduce-seq. The overlay vreset!/vswap!/volatile? drive a
volatile through ref-put!/:jolt/type (tagged-table only), so they're
overridden natively in post-prelude.ss.
driver.janet: drain each chez pipe to EOF instead of a single ev/read. A
program with a stdout side effect (newline) flushes in two writes and a
lone ev/read sometimes caught only the first chunk, dropping the trailing
value — an intermittent gate divergence. Now reads until the pipe closes.
Prelude parity 1886 -> 1898, 0 new divergences. Floor raised to 1898.
host/chez/multimethods.ss implements the multimethod runtime: defmulti/defmethod
expand to defmulti-setup/defmethod-setup calls (+ get-method/methods/
remove-method/prefer-method/prefers). A jolt-multifn record carries its dispatch
fn and a jolt=-keyed method table; jolt-invoke dispatches it (exact match, then
isa?/hierarchy with prefer-method, then :default), reusing the overlay's
isa?/derive/make-hierarchy. The multifn's ns comes from a runtime chez-current-ns
(default user; the prelude load sets clojure.core for print-method/print-dup).
Two emit-side changes were needed:
- late-bind (:late-bind-unresolved? ctx flag, default OFF): defmulti expands to a
bare-symbol setup call, so the analyzer doesn't intern the name and a forward
reference '(area ...)' after '(defmulti area ...)' in one form was 'Unable to
resolve symbol'. The strict compiler punts these to the interpreter; the Chez
back end has none, so the flag lowers an unresolved symbol to a var-ref against
the compile ns (open-world -e semantics). Set only by the Chez make-ctx /
jolt-chez; the main compiler keeps strict resolution (host_iface late-bind?
defaults nil).
- a :var call head now routes through jolt-invoke, since a late-bound var can hold
a multifn (or keyword/coll IFn), not just a procedure. Transparent for
procedures; the hot self-recursive call is a :local known-proc, stays direct.
Class-based dispatch ((class x)/String) deferred (needs deftype/class subsystem).
Parity 1506 -> 1530/2497, 0 new divergences. emit-test 302/302. Full janet gate
green (the analyzer flag is off there; suite flakiness under parallel load only).
Emit every non-macro clojure.core form through the live analyzer -> Chez emit
pipeline as a def-var! prelude (prelude mode, tier dependency order), load it
before a user expression, and you get an -e-capable jolt-chez: analysis on Janet,
execution on Chez. driver/emit-core-prelude assembles it (each form behind a
silent load guard so the Phase-2 multimethod forms don't break the rest);
bin/jolt-chez is the -e CLI, caching the prelude on disk keyed by source hash.
run-corpus-prelude.janet is the full parity gate this opens, the prelude-backed
sibling of run-corpus-chez. First baseline: 1220/2497 evaluated cases pass, 0 new
divergences (10 allowlisted: dynamic vars, class names, eval-order — deferred
Phase 2). The rest is the punch-list: ~360 emit-fail (real host interop, out of
the analyzer subset) and ~900 runtime crashes, mostly core fns calling
host-coupled seed natives with no Chez shim yet (str/format/vec, transients).
Follow-ups jolt-t6cr/kl2l/q3w8/9ls5.
Two shims landed to get the prelude to load and run. atoms.ss: atom/deref/swap!/
reset! (+ the compare/vals kernel) — needed at load time for
global-hierarchy = (atom (make-hierarchy)). predicates.ss: the type predicates +
name/namespace/boolean the overlay assumes are seed natives, matching the seed's
strict semantics. post-prelude.ss re-asserts char?/atom? after the prelude: the
overlay implements those by reading :jolt/type, which is false for Chez-native
chars/atoms, so its def-var! would clobber the correct native versions.
Per-case Scheme files are PID-unique so a foreground -e never reads a half-written
file while the gate runs.
Wire the real pipeline end to end: host/chez/driver.janet boots a compile-mode
jolt ctx, runs the EXISTING Janet-hosted analyzer on actual Clojure source to
real IR, feeds it to the Scheme emitter, and runs the result on Chez. Analysis
stays on Janet (the analyzer ports to Chez in Phase 2); execution is on Chez.
emit.janet now consumes live IR (pv/phm-normalized like the Janet backend) and
covers what the analyzer actually emits, not the hand-built inc-1 shapes:
- core ops arrive as :var clojure.core/+ etc., not :rt — lowered to native
Scheme via a native-ops table (mirrors backend.janet's), `=` to jolt=.
- var cells (host/chez/rt.ss): :def -> def-var!, :var -> var-deref. Late binding
so cross-var calls (run -> count-point) and the entry crossing resolve at use.
- named fns (defn / fn self-name) bind via letrec so self-recursion resolves.
- unsupported stdlib/host refs (no core on Chez yet) are rejected at EMIT time
(clean out-of-subset signal) instead of deref'ing to nil and failing at runtime.
Number model: jolt is all-doubles (no ratios; (/ 1 2) is 0.5), so literals emit
as flonums — matches the Janet host and keeps Chez out of exploding exact
rationals (mandelbrot). jolt-num->string prints integer-valued without ".0".
Two real bugs found via the corpus probe and fixed (regression rows added):
- loop bound in parallel (Scheme named-let) but Clojure loop is sequential — a
later init must see earlier bindings; wrap a let* around the loop.
- #(...) shorthand gensyms params with a trailing `#`, invalid in Scheme — munge
it to `_`.
Gate: test/chez/emit-test.janet runs the real analyzer -> Chez for (+ 1 2),
fib(30)=832040, mandelbrot run(40), and the two regressions, parity-checked
against the Janet oracle (6/6). First parity number via the new subset probe
(test/chez/run-corpus-chez.janet, JOLT_CHEZ_CORPUS=1): 182/182 compiled corpus
cases pass, 0 divergences; 2473/2655 out of subset pending core on Chez. Full
jpm/run-tests gate green (125 files). Chez tests skip cleanly without `chez`.
Perf note (unchanged plan): emitted fib(30) ~23ms vs hand-Scheme ~5ms — the
jolt-truthy? wrapper (~3x) plus flonum (not fixnum) arithmetic, both Phase-4
type-specialization levers.
