Two thread-safety bugs in the native FFI layer.
The HTTP server's accept/recv/send were plain foreign-procedures. A thread
inside a foreign call stays active for the stop-the-world collector, so the
accept loop sitting idle in accept() froze GC for the whole process whenever
another thread (a future, an async block) allocated. Mark the three blocking
calls __collect_safe so the thread deactivates for the call's duration —
collection proceeds while the accept thread waits. The args are an fd and
foreign-alloc'd buffers (outside the Scheme heap), so a collection mid-call has
nothing to move.
jolt.http-client built its -D header-file path from an unguarded (set! counter
(+ counter 1)) and counter mod 90000, with no per-process component. Concurrent
requests could compute the same path and clobber each other's headers. Use a
mutex-guarded monotonic counter plus the pid.
test/chez/ffi-server-test.ss exercises both (a (collect) while the server is
idle in accept(), temp-path uniqueness across threads, and a live request) and
is wired into the gate as `make ffi`.
deftype fields tagged ^:unsynchronized-mutable / ^:volatile-mutable can now be
reassigned in place from a method, as on the JVM. A jrec stores fields as cons
cells, so a new jolt-set-field! mutates the pair with set-cdr!. The deftype macro
rewrites (set! mutable-field v) in a method body to (set! (.-field inst) v), and
the analyzer compiles a (set! (.-field obj) v) target to jolt-set-field! — so
both the rewritten symbol form and an explicit interop (set! (.-root this) v) go
through one path. Field reads remain a snapshot at method entry, which is correct
for the universal read-then-set pattern (a repeated set! of the same field in one
call would read the entry value).
Closes the set!-of-local SCI failures: SCI load 202 -> 205/218.
Found in a read/eval review: a local named like a special form wrongly took
over operator position. (let [if (fn ...)] (if true 1 2)) returned the fn, but
per spec section 3 (and the reference) special-form heads are not shadowable;
only macros are. Two fixes: drop the (not shadowed) guard on the special-form
branch of analyze-list (so an (if ...) head is always the special), and prefix
a local whose name is a Scheme keyword when emitting (so a value local legally
named if does not shadow the (if ...) the back end emits). Value-position
locals named if/or/case still work.
(type r) returned a symbol user.TyR, so (= (symbol (str (type r))) (type r))
was true; the JVM's type is a Class (not a Symbol) so it's false. jolt models
classes as strings, so a record's type is now its ns-qualified class-name
string — equal to (class r), as on the JVM where type and class coincide for a
record. The symbol-keyed print-method defmethods already fall through to the
default record printing, so they're unaffected. Closes type-of-record.
(def ^String tv ...) left (:tag (meta (var tv))) as the unresolved "String";
the JVM compiler resolves the hint to java.lang.String at def time. Add a
resolve-class-hint host seam (built from the existing class-token table) and
resolve a def's :tag through it in the analyzer. The reader path
(read-string "^String x") stays unresolved, matching the JVM (only the
compiler resolves). Closes ^Type-tag-on-var.
- (class x) returns per-type JVM class names (Long/Double/Ratio/Character/Atom),
not a blanket java.lang.Number.
- register fully-qualified class tokens (java.lang.Long, clojure.lang.Keyword,
clojure.lang.Atom, ...) that self-evaluate to their name, so (= (class 1)
java.lang.Long) and (instance? clojure.lang.Atom x) resolve.
- instance? recognizes Long/Double/Ratio/Character/Symbol/Atom/IFn built-ins.
Closes class number/string/keyword/name, instance? Atom, atom?. Corpus 2699->2705.
- == 1-arg returns true for any value (Clojure short-circuits before the number
check), not 'requires numbers'.
- current-time-ms wired to now-millis so the time macro works.
- subvec truncates float/ratio indices via long (Scheme quotient rejects flonums).
- defonce checks bound? not var-get — in a top-level do the name is already an
unbound interned cell, which var-get throws on.
- drop the line-seq corpus row (used janet/spit, N/A); allowlist char-array
(needs Class/forName "[C").
Corpus 2678->2683, floor raised. Re-minted. Full gate green; CI green.
jolt-cf1q.7
Remove the 17 rows that exercised the Janet FFI bridge (interop/janet bridge,
interop/jolt.interop) and the Janet build-time env scrub (host-interop/bake env
scrub, janet.os/setenv) — none exist on any non-Janet host, so they only added
crash noise. Portable interop is covered elsewhere. corpus 2920->2903 rows;
gate 2678/2742 0 new div, certify 0 new/0 stale.
Rewrite the README, CLAUDE.md build/architecture sections, test/chez/README,
and conformance SPEC for the Janet-free world: bin/joltc + make test, the
self-hosting bootstrap, the frozen JVM-sourced corpus. CI installs Chez + JDK/
Clojure and runs 'make test' (was Janet/jpm).
jolt-cf1q.6
Remove the Janet seed (src/jolt/*.janet: reader, value layer, vars/ns, the
tree-walking interpreter, the Janet backend, the optimizing compiler), the
Janet->Scheme cross-compiler (host/chez/{driver,emit,jolt-chez}.janet),
bin/jolt-chez, the jpm build (project.janet) and the Janet test runner
(run-tests.janet), plus the entire Janet test suite. jolt now builds and runs
on Chez alone: bin/joltc off the checked-in seed, bootstrap.ss to rebuild it.
The portable Clojure stays: jolt-core/**, host/chez/**.ss, and the stdlib +
tooling under src/jolt/clojure + src/jolt/jolt (read by the seed build, no
Janet). The gate is 'make test' (self-host, corpus, unit, cli smoke, certify).
Drop the sci and clojure-test-suite submodules (used only by deleted Janet
integration tests); irregex stays.
Filesystem corpus/unit cases that probed project.janet now probe README.md.
jolt-cf1q.6
Add a Janet-free gate so correctness can be judged with only Chez + Clojure:
- host/chez/run-corpus.ss: corpus.edn vs JVM expecteds, lifting the per-case ns
isolation from the old Janet driver; reads corpus.edn via the jolt reader.
- host/chez/run-unit.ss + test/chez/unit.edn: the host-specific unit cases,
evaluated in-process and compared to baked expecteds.
- host/chez/selfcheck.sh: self-host fixpoint (bootstrap.ss rebuild == checked-in seed).
- host/chez/smoke.sh: real bin/joltc CLI smoke.
- host/chez/remint.sh: re-mint the seed to a byte-fixpoint after a source change.
- Makefile: 'make test' runs the lot; 'make remint' rebuilds the seed.
Numbers match the Janet gate: corpus 2679/2757 0 new div, unit 450/450, certify
0 new/0 stale.
jolt-cf1q.6
The unit tests in test/chez/_*.janet now drive bin/joltc (the zero-Janet
spine) and judge against baked expected values instead of a live build/jolt
run. Ten of them captured the oracle from build/jolt per case; their values
are now literals (one env-dependent javastatic case became a predicate so it
stays portable). The rest already had literal expecteds with a redundant
build/jolt sanity check, now dropped.
Retire emit-test/emit-parity/reader-parity: they compared the Chez/Clojure
path against a live Janet evaluation, emitter, or reader. That migration check
is done, and run-corpus-zero-janet (Chez analyzer vs the JVM corpus) plus
certify.clj cover correctness now.
Rewrite the README for the current zero-Janet gate.
jolt-5oci
run-corpus.janet drove a target binary (default build/jolt, the Janet host)
through the corpus and run-corpus-chez.janet was the all-flonum subset probe.
Both compare against corpus.edn, which is now JVM-sourced, so the all-flonum
hosts diverge on every numeric/host case. The zero-Janet spine gate
(run-corpus-zero-janet.janet) plus certify.clj are the oracles; drop these and
the stale test/chez/known-divergences.edn allowlist they used.
corpus.edn :expected is now the value reference JVM Clojure produces, set by the
new test/conformance/regen-corpus.clj (one JVM process, per-row thread watchdog).
167 rows moved to the JVM value: ratios (/ 1 2)=>1/2, doubles (double 3)=>3.0,
shared-heap concurrency (the future/pmap/agent cases), clojure.math doubles. The
JVM is the spec; jolt is measured against it.
known-divergences.edn shrinks to the rows whose JVM value is an opaque host object
that can't round-trip to source (Java arrays, transients, atoms, beans, proxies,
chunks all print as #object[..@addr]) plus (fn* foo) and a few racy concurrency
cases (:flaky). The zero-janet gate's allowlist becomes the set of host gaps vs the
JVM spec (no Class/array/BigDecimal, :jolt reader, jolt's own printing).
Math/clojure.math sqrt/pow/floor/trig now return doubles (Chez returns exact for
exact args, e.g. (sqrt 9)=>3); JVM always returns a double.
extract-corpus.janet no longer writes corpus.edn unless asked (the test runner
imported it and was silently overwriting the JVM corpus with the spec sources'
placeholder answers). The prelude parity gate is deleted — the zero-janet spine +
certify.clj are the oracles.
zero-janet 2678 (0 new divergences), certify 0 new / 0 stale, emit-test 330/330.
jolt was all-flonum (one :number type, inherited from Janet whose only number
type is a double). The Chez runtime has a full numeric tower, so the zero-Janet
path now carries it = JVM Clojure semantics:
(/ 1 2) => 1/2 (exact Ratio, was 0.5)
(integer? 3) => true (integer? 3.0) => false (float? 3.0) => true
(ratio? (/ 1 2)) => true (= 3 3.0) => false (== 3 3.0) => true
(+ 1 2) => 3 (exact) (/ 1.0 2) => 0.5 (double)
jolt= was already exactness-aware (values.ss) and == is value-equality, so
=/== match the JVM split. The reader preserves exactness (integer literals exact,
a/b ratios exact rationals, decimals/exponents flonums); backend_scheme emit-const
renders exact ints/ratios and flonums faithfully; the value-position arithmetic,
count, int, compare, bit ops, parseLong, string .length/.indexOf, range,
timestamps, and array bytes return exact integers (= JVM int/long) instead of
coercing to flonum. double/parseDouble/clojure.math floor|ceil|signum stay double.
Only the zero-Janet path carries the tower (the Janet reader loses exactness into
a double before emit). The prelude/all-flonum path is unaffected for compiled code;
the runtime reader is shared, so a couple of all-flonum reader assertions become
value (==) assertions. ~16 numeric corpus cases now give the JVM tower value vs the
Janet-era :expected and are allowlisted as tower divergences (Chez == reference
JVM) pending the corpus flip to JVM (jolt-ecz0). No BigDecimal type (1M).
Re-minted. zero-janet 2682 (floor 2698->2682, the reclassified tower cases), 0 new
divergences; fixpoint 10/10, bootstrap 6/6, spine 35/35, cli 49/49; Janet gate 155
files 0 failed.
Two Chez reader bugs, both JVM-parity gaps:
inc'/+'/foo' (trailing apostrophe) were mis-read as a symbol followed by a
quote macro, because the reader treated ' as a terminator. In Clojure ' is a
NON-terminating macro char (constituent after the first char). Since the seed
is minted on Chez, (def inc' inc) became (def inc 'inc), clobbering inc's var
cell with its own symbol -- so (var-get (var inc)) returned the symbol, not the
fn. Drop ' from the token terminator set; a leading ' still quotes.
^bytes [b] / ^String [x y] return-type hints: the Chez reader lowered ^meta on
a collection to a (with-meta vec meta) form, but emitted a QUALIFIED
clojure.core/with-meta while the Janet reader emits a bare with-meta -- so the
fn/defn macros' unwrap logic (matching the bare head) slipped past it and choked
on a non-vector arglist. Emit bare with-meta to match Janet, and unwrap a
(with-meta <vec> _) arglist in analyze-fn as a backstop.
Re-minted the seed. zero-janet 2699, prelude 2652, Janet gate 155/0, fixpoint
10/10, bootstrap 6/6, all 0 new divergences.
realized? threw 'not supported on' for a jolt-lazyseq record (the overlay
reads :jolt/type); add a jolt-lazyseq? arm to the post-prelude wrapper
reading the record's own realized? flag.
conj! 1-arity (conj! coll) is the transducer-completion arity and returns
coll as-is on the JVM, no transient check — we threw 'not a transient'.
Both gates: zero-janet 2696->2698, prelude 2649->2652, 0 new divergences.
Audit of the janet.*/jolt.interop/STM corpus cases vs Chez equivalents: the Janet
FFI-bridge cases (janet/string, janet/type, janet.math/sqrt, janet.string/ascii-
upper) test functionality already covered by PORTABLE corpus cases (str 29, type 9,
sqrt 3, upper-case 9), so they're safe to delete in Phase 5. Two needed a Chez
equivalent so they pass instead of being lost:
- clojure.lang.LockingTransaction/isRunning -> false (no STM on jolt).
- line-seq: the corpus case used janet/spit setup but was the SOLE line-seq
coverage (0 other cases). Ported janet/spit->spit, and added a native Chez
line-seq (drain a jhost io/reader + split on newline; no trailing empty line)
that delegates a Janet map-reader to the overlay version.
zero-Janet 2694->2696, prelude floor 2648; self-host + Janet gate + JVM cert green.
jolt-cf1q.7 jolt-0obq
hc-list? required cseq-list? (a reader-built list), so a form built at runtime via
concat/map/cons — a lazy cseq with list?=#f — was rejected as "unsupported form".
In Clojure any seq is a valid call form, so accept any cseq. Lights up macros that
build their expansion with concat/list and (eval seq-form).
zero-Janet 2692->2694, 0 new divergences; self-host + Janet gate + JVM cert green.
jolt-cf1q.7
Date/time (inst-time.ss): java.util.Date / java.sql.Timestamp ctors accept ms or
another date value (ms-of) -> a jinst; java.text.SimpleDateFormat (pattern + .format
via the existing format-ms UTC engine; .setTimeZone accepted); java.util.TimeZone/
getTimeZone. instance? answers Date true / Timestamp false for a jinst (a Date is
not a Timestamp on the JVM).
clojure.edn/read over a reader (io.ss + post-prelude): the overlay edn.clj's
drain-reader is janet/type-coupled, so Chez drains the jhost StringReader/
PushbackReader to a string and reads the first EDN form. Unblocks jolt-uicd.
Native Chez throughout (no vendoring): Chez date arithmetic + string ports. zero-Janet
2688->2692, 0 new divergences; self-host + Janet gate + JVM cert green.
jolt-cf1q.7 jolt-dcmm jolt-7t3l jolt-uicd
deftype binds the type name as a VAR (the make-deftype-ctor closure), but (P. 5)
lowers to (host-new "P"), which only checked class-ctors-tbl -> "No constructor".
Fall back to resolving the class name as a var in the current ns / clojure.core
and invoking it — so (P. args) constructs the same jrec as the ->P factory, and
protocol method dispatch (.m / .-field) over it works.
zero-Janet 2685->2688 (no-constructor 5->2); prelude floor bumped to 2641 (the
delay batch's run). Self-host + Janet gates + JVM cert green.
jolt-cf1q.7
Add a thread-safe delay type (concurrency.ss): make-delay wraps a thunk; deref/
force run it once under a lock and cache (JVM delays are thread-safe + memoized).
delay? and realized?-on-a-delay are native; the overlay's `delay` macro
(-> make-delay) and `force` (-> deref) now work. realized? wrapper (post-prelude)
and the deref chain gain a delay arm. Removed the np-delay? stub from
natives-parity.ss (the real type lives in concurrency.ss).
Seed unchanged (no re-mint). zero-Janet 2673->2685, 0 new divergences; Janet gate
+ JVM cert green.
jolt-cf1q.7
The zero-Janet runner wrapped each case as (= EXPECTED ACTUAL) and checked the
result was true. That nests ACTUAL's top-level (do ...), so a case like
(do (defmacro m ...) (m 1)) can't use the macro it just defined — the analyzer
punted defmacro -> "uncompilable" (35 cases).
Match certify.clj's eval-isolated instead: carry EXPECTED and ACTUAL as separate
sources and evaluate ACTUAL as its own top-level program (jolt-compile-eval unrolls
the top-level do, so a macro defined earlier is usable later), then compare to
EXPECTED with =. Evaluating ACTUAL from SOURCE (not (eval (quote A))) preserves the
reader's map-literal source order, so the eval-order cases still pass.
eval-corpus-zero-janet / program-corpus-zero-janet now use a 3-field TSV
(label/expected/actual); run-corpus-zero-janet's per-case debug path evals both
sides too. Only run-corpus-zero-janet uses these (the prelude gate is untouched).
zero-Janet 2642->2673 (analyzer "uncompilable" 35->4); 1 new allowlisted divergence
(`{:a ~x :b ~y} syntax-quote map construction doesn't preserve source eval order —
pmap is unordered). Janet gate + JVM cert green.
jolt-cf1q.7
host/chez/natives-array.ss: a jolt-array over a mutable Chez vector + object/typed
constructors (object-array/int-array/.../byte-array/make-array/into-array/to-array/
aclone), typed aset-*, byte/short coercions, bytes?/bytes/ints/..., and eager
chunk-buffer/chunk-* (Jolt doesn't chunk). count/nth/seq/get and jolt.host/ref-put!
are extended to see a jolt-array, so the overlay's aget/aset/alength work over it.
Numbers it produces are flonums (jolt's rep) so exactness-aware = holds. char-array
stays in io.ss (a char-SEQ that io/reader/str/slurp consume).
natives-parity.ss adds: __reader-features / -set!, reader-conditional, re-matcher,
delay? (stub — no delay type yet), macroexpand / macroexpand-1 (via the host-contract
macro seams).
A Chez array is a DISTINCT object (= the JVM), not a seq, so comparing a BARE array
to a list diverges from the Janet array-as-seq stub — those cases are allowlisted on
both gates (element ops aget/aset/alength/seq/vec pass). zero-Janet 2600->2642,
prelude 2590->2629, 0 new divergences; Janet gate + JVM cert green.
jolt-cf1q.7
Native Chez shims for clojure.core fns that live in the Janet seed but had none
on the zero-Janet spine (they resolved to nil -> "not a fn"):
- host/chez/natives-parity.ss: hash / hash-combine / hash-ordered-coll /
hash-unordered-coll (24-bit masked like core_extra), transient?, rseq (vectors +
sorted colls), cat (transducer).
- jolt-invoke now dispatches a TRANSIENT vec/map/set as a fn (callable on the JVM):
((transient [10 20 30]) 1) -> 20.
- ns.ss: ns-resolve, ns-imports, remove-ns, intern, alias, ns-unalias, refer,
ns-refers, refer-clojure, alter-meta!, reset-meta!, and a real ns-aliases (was a
stub returning {}).
- runtime (require ...)/(use ...) now register :as/:refer into the Chez ns tables
(was a no-op). The Chez analyzer already pre-registers at analyze time, but when
the JANET analyzer compiled the form (prelude path) the Chez tables stayed empty,
so ns-aliases/ns-resolve over an alias diverged — this fixes both paths.
Seed unchanged (overlay doesn't reference these at mint time). zero-Janet corpus
2567->2600, prelude 2557->2590, 0 new divergences on either; Janet gate + JVM cert
green. Filed jolt-vgrp for the pre-existing var-get-of-scalar-native-op quirk.
jolt-cf1q.7
Mirrors test/bench/core-bench.janet's 8 compute programs + methodology (load the
runtime once, time compile+run of each, min of N) so the zero-Janet Chez path is
comparable to the Janet compile path.
Result: Chez is 3-33x faster than Janet across the set (~320ms vs ~5000ms total,
~15x), biggest on seq/map/reduce, smallest on fib (3.2x). So deleting Janet is a
perf win, which satisfies the "perf confirmed" precondition Phase 5 gates on. Chez
is ~2-28x slower than JVM Clojure, expected and not the bar (and the Janet-only
optimizing modes haven't been tried on Chez).
jolt-cf1q.5
Replace the Janet synchronous agent shim (agent = atom, send applies inline) with
JVM-style async agents: send/send-off enqueue an action and a single worker thread
per agent applies them in order; deref reads the current (maybe not-yet-updated)
state without blocking; await blocks until the queue drains. A validator rejection
or a thrown action puts the agent in an error state (agent-error) and halts the
queue; restart-agent clears it. send and send-off share one serialized worker (a
superset of the JVM's fixed/cached pool split). Native versions re-asserted in
post-prelude over the overlay; await/restart-agent are new.
Corpus: the two "send/send-off applies" cases do (send a f) (deref a) with no
await, so they now read state before the action runs — diverging like the JVM
(the suite was literally "synchronous shim"). Allowlisted on both gates; floors
-2 (zero-Janet 2569->2567, prelude 2559->2557). cli-test covers async agents via
await (ordered 100-send dispatch, error capture) — 49/49. Janet gate + JVM cert
green; 0 new divergences on either corpus.
jolt-byjr
No mature Chez fibers library exists and this is a threaded Chez build, so a go
block is an OS thread and a channel is a mutex+condition blocking queue: <! / >!
are the blocking <!! / >!! and work anywhere (no CPS transform), like the Janet
stackful-fiber model but with real parallelism and a shared heap.
host/chez/async.ss provides chan (unbuffered rendezvous / fixed / dropping /
sliding), <! >! <!! >!! close! alts! timeout put! take! buffer ctors, channel
transducers, and go-spawn, all def-var!'d into clojure.core.async; go/go-loop/
thread are macros (mark-macro!) expanding to go-spawn, mirroring src/jolt/
async.janet. Binding conveyance rides the thread-parameter binding stack from
pt.1. alts! polls with a 1ms backoff (no cross-channel wait-set yet) and is
take-only, matching the Janet impl.
(require '[clojure.core.async ...]) resolves it with no file load — the vars are
resident and require just registers the :as/:refer.
cli-test covers go/buffered-drain/nested-<!/alts!/transducer/timeout/binding-
conveyance (43/43). core.async isn't in the conformance corpus (feature-gated
:async/core-async), so coverage is the Chez cli-test plus the existing Janet
core-async-spec. Seed unchanged (no .clj touched). Prelude corpus 2534->2559,
zero-Janet 2569, 0 new divergences on either; Janet gate + JVM cert green.
jolt-byjr
future/future-call run the body on a native thread (fork-thread) over the SAME
heap — JVM semantics, not Janet's isolated-heap snapshot. deref blocks on a
mutex+condition latch; timed (deref f ms val) uses an absolute deadline.
promise is a real blocking promise (deref parks until deliver), replacing the
Janet non-blocking atom shim. future?/future-done?/future-cancelled?/future-cancel
/realized? are native (the overlay versions read Janet map keys); re-asserted in
post-prelude over the overlay. pmap/pcalls/pvalues (overlay, over future) light
up for free.
Thread-safety this forces:
- atoms get a per-atom mutex; swap!/swap-vals! are a JVM-style CAS loop (f runs
outside the lock, so a watch/validator can deref the same atom); reset!/
compare-and-set! are atomic.
- the dynamic binding stack becomes a Chez thread-parameter, so each future/thread
has its own; Chez inherits it at fork, giving binding conveyance (the shim also
installs an explicit snapshot).
- Thread/sleep really sleeps now (a worker sleeping doesn't block the parent).
Re-minted the seed: future-call now resolves at compile time, so pmap compiles to
a var-deref instead of the host-static-call fallback that crashed. image.ss
unchanged.
Corpus: the 2 snapshot cases now match the JVM (shared) not Janet (isolated) —
allowlisted on both Chez gates; the two racy future-cancel cases allowlisted;
"promise undelivered" (blocks on JVM/Chez, profile :bucket :timeout) skipped like
:throws. Zero-Janet corpus 2544 -> 2569, 0 new divergences, floor raised. Full
Janet gate + JVM cert green.
jolt-byjr
The compiler image is already resident at runtime on the Chez spine, so eval
and load-string are just wiring: make them clojure.core functions instead of
analyzer special forms.
- eval / load-string are now functions, not special forms. Dropped "eval" from
the host-contract special-symbol lists so it resolves as an ordinary var, and
def-var! both in compile-eval.ss. eval takes an already-read form (e.g. from
quote/list) and compiles+evals it in the current ns; load-string reads every
form from a source string and evals each, returning the last.
- Runtime defmacro: jolt-compile-eval-form intercepts a (defmacro ...) form
before analysis, defs the expander fn + mark-macro!s the var, exactly as
emit-image.ss does at build time. The two helpers (macro-form? / defmacro->fn)
move to compile-eval.ss and emit-image.ss reuses them.
- Top-level (do ...) is now unrolled form-by-form, like Clojure, so a defmacro
or def in an earlier subform is visible (macro flag set / var interned) before
a later subform is analyzed. This is what makes multi-form -e with a macro work.
Seed is byte-identical (no source references eval), so no re-mint; bootstrap-test
still passes. Zero-Janet corpus 2534 -> 2544 (eval/load-string cases now run),
0 new divergences; floor raised. Prelude corpus, JVM cert, full Janet gate green.
jolt-r8ku
The corpus certifier (test/conformance) flagged four cases where jolt's
hand-written :expected matched a real defect rather than Clojure. Fixed in the
jolt-core overlay, corrected the spec :expected, re-certified against JVM Clojure:
- ex-message: returns nil for a non-throwable (dropped the lenient string branch);
still returns the message for ex-info. (jolt-l8e8)
- munge: preserves the argument's type — a symbol munges to a symbol, not a string.
(jolt-hc35)
- print: (print nil) emits "nil", not "" (top-level nil guard; str yields "").
(jolt-pqio)
- bounded-count: uses the counted? fast path (full count), else counts up to n via
seq — was (min n (count coll)), wrong for counted colls. Added an uncounted-coll
spec case. (jolt-2507)
Removed the 4 :bug entries from known-divergences.edn (now certified), regenerated
corpus + profile, re-minted the Chez bootstrap seed (clojure.core changed). Gates:
Janet 155/0, JVM certify clean, both Chez corpus gates 2534 (floors raised),
bootstrap 6/6, fixpoint intact.
Makes the host-neutral corpus a first-class language specification with
conformance levels, not just a regression suite.
- [suite label] is now a unique, stable case id (extract-corpus disambiguates
duplicate labels with ' (N)' — one collision existed).
- certify.clj --profile emits test/conformance/profile.edn: every non-portable
case classified by the host feature it requires (numerics/double-only,
concurrency/snapshot, host/jvm-interop, host/arrays, host/janet,
async/core-async, runtime/eval, reader/jolt, printer/jolt, strictness/jolt,
impl/representation, bug). 2670 of 2919 cases are portable (pass on any faithful
Clojure); 249 are feature-gated.
- SPEC.md documents the contract: row schema, the JVM oracle, conformance levels,
the feature vocabulary, and a worked new-runtime harness — so hosting jolt
elsewhere and proving it correct is read-one-file mechanical.
Janet gate 155 files 0 failed; certify + zero-janet gates green.
The corpus only ever saw test/spec/*-spec.janet; the 355 hand-written cases in
test/integration/conformance-test.janet (inline Janet, the lazy-seq / IFn /
destructuring / transducer essentials) were invisible to it and to any non-Janet
runtime. extract-corpus.janet now also pulls that (def cases ...) vector, deduped
by :actual, organized into 41 'conformance / <section>' suites recovered from the
file's ### headers. Corpus 2658 -> 2919 rows (+261 unique).
JVM certification: only 1 new divergence ((/ 2) => 0.5 vs 1/2, the all-double
numeric model) — classified. Chez gates: +1 known host gap (instance? Atom, atom
class identity, Phase 4) allowlisted in both runners; parity rose 2295 -> 2533 on
both, floors raised. Janet gate 155 files 0 failed; certifier green (0 new/stale).
Deferred: 41 non-literal core-async spec rows ((a "src") async-harness wrapper)
need harness context the corpus format doesn't carry — left for inc3.
The corpus carried hand-written :expected values — a regression suite but a weak
spec (it checked jolt against its authors, not against Clojure). certify.clj runs
every corpus row's :actual and :expected through reference JVM Clojure 1.12.5 (fresh
user ns per case, output/stdin sunk, 5s per-case watchdog) and compares with =.
Result: of ~2487 vanilla-certifiable rows, 2416 match real Clojure exactly. The 71
divergences are all classified in known-divergences.edn — mostly deliberate
jolt-specific/host-model deltas (all-double numerics, snapshot concurrency, no-JVM
host model, jolt reader features, printer, strictness), plus 4 genuine bugs filed
as beads (jolt-l8e8 ex-message, jolt-hc35 munge, jolt-pqio print-nil,
jolt-2507 bounded-count).
certify-test.janet gates it: skips without clojure on PATH, else fails only on a
NEW (unclassified) divergence or stale allowlist entry; flaky timing-dependent
cases (future-cancel) tolerated either way. Full gate 155 files 0 failed.
The runtime counterpart to bootstrap.ss. host/chez/cli.ss loads the checked-in
seed + the zero-Janet spine and compiles+evals a -e expression entirely on Chez;
bin/joltc execs it. With the seed checked in, a clone runs jolt with only Chez
installed — no Janet at build or run time. Multi-form -e wraps in (do ...) to
match Clojure. test/chez/cli-test.janet 9/9.
Makes the inc8 fixpoint the actual build. host/chez/bootstrap.ss loads a seed
(prelude, image) pair and rebuilds the clojure.core prelude + compiler image from
source via the on-Chez compiler — read/analyze/emit all on Chez, zero Janet.
The seed (host/chez/seed/{prelude,image}.ss) is the checked-in bootstrap
compiler, minted once via the fixpoint (driver/mint-chez-seed* iterates
bootstrap.ss from the Janet-emitted pair to a joint byte-fixpoint). It's a joint
fixpoint: rebuilding from an up-to-date seed reproduces it exactly. So a fresh
checkout + Chez (no Janet) yields a working jolt.
test/chez/bootstrap-test.janet spawns only chez, asserts the rebuilt artifacts
match the seed byte-for-byte and compile+run real cases. Drift (seed sources
changed) fails the test with a re-mint pointer; host/chez/seed/README documents
re-minting.
Extends the fixpoint beyond the compiler image to the whole emitted system.
emit-image.ss now handles macros (defmacro -> bare expander fn + def-var! +
mark-macro!) and re-emits the clojure.core prelude (all tiers + stdlib) on Chez
via jolt-emit-prelude; driver's emit-image-on-chez takes an emit-fn arg.
The prelude converges at pstage3==pstage4 (one stage later than the compiler's
stage2==stage3) because macro expanders bake an auto-gensym id at emit time, so a
Janet-emitted macro carries a different id than a Chez-emitted one — only once
both stages load a Chez-emitted prelude does it stabilize.
fixpoint-test now proves: compiler stage2==stage3, prelude pstage3==pstage4, and
the fully Chez-emitted system (Chez prelude + Chez image, no Janet artifact in the
loop) compiles+runs real cases. 10/10.
The zero-Janet spine proves the on-Chez analyzer/emitter compile arbitrary
Clojure faithfully. This proves the stronger property: the on-Chez compiler
reproduces itself. emit-image.ss re-emits the compiler sources (jolt.ir +
jolt.analyzer + jolt.backend-scheme) ON CHEZ via the loaded image; feeding it
stage1 (the Janet cross-compile) yields stage2, feeding stage2 yields stage3.
stage2 and stage3 are byte-for-byte identical, and stage2 is a working compiler
(real cases compile+run through it). stage1 differs from stage2 only in gensym
numbering, so the fixpoint is stage2==stage3.
driver: emit-image-on-chez / program-emit-image spawn a fresh chez per stage
(clean gensym state). test/chez/fixpoint-test.janet gates it (skips without chez).
(require '[clojure.string :refer [blank?]]) then an unqualified blank? now
resolves. chez-register-spec! registers :refer names (in addition to :as) into a
refer table; hc-resolve-cell's unqualified branch consults it before clojure.core.
Zero-Janet corpus parity 2293 -> 2295 = the Janet-hosted oracle's exact pass
count. The self-hosted Chez compiler (read -> analyze -> emit -> eval, no Janet)
now compiles every corpus case the Janet-hosted compiler does, with 0
divergences. Remaining failures are shared runtime breadth (host interop,
futures, runtime eval) deferred to Phase 4 / jolt-r8ku. Floor 2295.
The batched zero-Janet runner wrote one case per TSV line, so a multi-line
source (e.g. a ;comment\n inside a map literal) split the line and the case was
truncated -> a false "apply non-procedure" crash. Escape \n/\t/\\ when writing
the cases file and unescape in the runner before eval.
Zero-Janet corpus parity 2288 -> 2293 (the 5 comment-in-map cases), 0
divergences — now within 2 of the Janet-hosted oracle (2295). Floor 2293.
Reader gaps the Chez-hosted analyzer hit where the Janet reader didn't:
- ##Inf / ##-Inf / ##NaN symbolic literals (## dispatch -> flonum).
- #(...) anonymous fn shorthand -> (fn* [p__N#] body), with % / %N / %& and the
max-positional arity rule; scans + rewrites list/vector/set/map bodies.
- #?(...) reader conditional: feature set {:jolt :default}, first matching clause
wins. #?@ splicing not yet supported (one niche case allowlisted).
- (ns name (:require [a :as x])) — the require pre-scan now also reads aliases
from an ns form's :require/:use clauses, not just bare (require ...).
Zero-Janet corpus parity 2240 -> 2288, 0 divergences (2 now-reachable cases
allowlisted: str of Infinity inside a collection — same as the prelude gate —
and #?@ splice). spine-test 35/35; prelude parity 2295 unchanged, 0 new
divergences.
(require '[clojure.string :as s]) then s/foo crashed "Unknown class s": the
Chez analyzer resolved a qualified symbol's ns literally, and there was no
alias table (ns.ss jolt-ns-aliases was a stub, require a no-op). Add an alias
table + chez-register-spec! (parses [ns :as a]) in ns.ss; hc-resolve-cell now
resolves a qualified ns through it; compile-eval pre-registers a form's
require/use :as aliases before analysis (analysis precedes the runtime require,
so the runtime require staying a no-op is fine). The batched gate runner clears
the alias table between cases.
Zero-Janet corpus parity 2159 -> 2240, 0 divergences. spine-test 35/35.
Point the Chez-HOSTED analyzer at the full parity corpus (read -> analyze ->
emit -> eval, all on Chez, no Janet) and close the divergences so the
self-hosted compiler is faithful: 0 divergences, 2159/2494 pass.
Keystone: the on-Chez emitter ran with prelude-mode off, so every call to a
non-native clojure.core fn tripped the "unsupported stdlib fn" out-of-subset
guard. The zero-Janet spine always has the full prelude loaded, so turn
prelude mode on in compile-eval.ss (22% -> 84% pass on a sample).
Faithfulness fixes (each was the Chez host/reader diverging from the Janet
analyzer; fixed in the keeper, not the seed):
- emit-const read a char's codepoint via (get v :ch) — the Janet rep; on Chez a
char is native. Route through a new form-char-code host-contract fn (41 cases).
- next over a lazy seq returned the empty-list terminator (truthy), not nil, so
butlast and other (if (next s) ...) loops ran one step too far — broke
some->/some->>/cond->>.
- reader: radix literals (2r1010/16rFF/36rZ), #^ deprecated metadata, ^meta on
collections (lowers to a runtime with-meta form like the Janet reader),
map-literal source order (values eval left-to-right), and nested syntax-quote
over a literal collapses at read time.
- keyword "a/b" splits into ns/name like the seed (destructure {:keys [x/y]}).
- form-syntax-quote-lower implemented on Chez (was a throwing stub).
7 divergences allowlisted: the same print-method-multimethod / host-class set
the prelude gate defers. 328 crashes remain = shared runtime breadth (host
interop, missing core fns, eval/load-string) deferred to Phase 4 / jolt-r8ku,
not compiler faithfulness.
Gate + speedup: test/chez/run-corpus-zero-janet.janet (floor 2159). Its batched
runner (driver/eval-corpus-zero-janet) runs every case in ONE chez process —
load the runtime once, guard + reset the user namespace per case — instead of a
fresh process per case: 1379s -> 1.6s.
spine-test 35/35; Janet gate 151/0; prelude parity 2295/2494 unchanged, 0 new
divergences.
The on-Chez analyzer (inc6a) skipped macros, so let/when/->/defn didn't
expand from source. Each core/stdlib defmacro now emits into the prelude as
(def-var! ns name <expander fn>) + (mark-macro! ns name); form-macro?/
form-expand-1 on Chez look up the macro flag (rt.ss var-macro-table) and
apply the expander to the unevaluated arg forms, and the analyzer re-analyzes
the result. The expander's syntax-quote template was lowered to construction
code at cross-compile time, so it builds the expansion via __sqcat/__sqvec/
__sqmap/__sqset/__sq1 (new host/chez/syntax-quote.ss) as Chez reader forms.
Emit the bare (fn ...), not (def NAME (fn ...)): analyzing a def would
host-intern! NAME as a non-macro stub in the build ctx, and that stub makes a
later (require '[stdlib-ns]) skip loading the real macro — with-pprint-dispatch
then resolved as a fn and returned its unexpanded template. Wrapping the
lambda in def-var! manually never interns NAME. Fuller build-ctx isolation
(so stdlib cases pass instead of crash) tracked in jolt-lpvi.
__sqset builds a real set VALUE, not the reader's tagged-set form — a runtime
`#{~@xs} must be a set, not a map. form-set? additionally recognizes a pset so
a macro template's #{...} expansion still re-analyzes as a set literal.
spine-test 35/35 (20 macro cases: when/when-not/let/->/->>/and/or/cond/if-not/
defn run zero-Janet from source). Prelude parity 2280->2295, 0 new divergences.
Full Janet gate green.
Cross-compile jolt.ir + jolt.analyzer + jolt.backend-scheme to Scheme def-var!
forms via the existing Janet emit pipeline (driver/emit-compiler-image) and run
them ON CHEZ over a Scheme jolt.host impl. A macro-free Clojure expression now
compiles and runs with no Janet in the loop: read (reader.ss) -> analyze
(jolt.analyzer on Chez) -> IR -> emit (jolt.backend-scheme on Chez) -> eval.
host/chez/host-contract.ss is the jolt.host contract on Chez (the portable seam
the cross-compiled analyzer/emitter call): form-* over the Chez reader's forms,
resolve-global/compile-ns/host-intern!/late-bind? over the var-cell registry. ctx
is an opaque record carrying the compile ns. Native-op names are declare-var!'d
into clojure.core so +, *, <, ... classify as :var and the emitter's native-op
path lowers them. form-macro? is a #f stub and macro expansion / syntax-quote /
record hints are stubs for inc6b (runtime macros, jolt-r8ku).
host/chez/compile-eval.ss is the spine entry (read-string -> analyze -> emit ->
eval). driver gains emit-compiler-image / ensure-compiler-image (image caching)
and program-zero-janet / eval-zero-janet.
Two bugs fixed in the keeper, not reproduced:
- the Chez reader stores an unqualified symbol's ns as #f, but the analyzer tests
(nil? ns); hc-sym-ns normalizes #f/'() -> jolt-nil. Without it every handled
special (if/do/fn*) misanalyzed as a plain invoke.
- char (int->char) was missing from clojure.core on Chez; the emitter's
chez-str-lit needs it for keyword/string consts. Added jolt-char to converters.ss.
Gate: test/chez/spine-test.janet 15/15 (Chez-hosted analyzer value == Janet-hosted
oracle through the same emitter/RT; only the analysis host differs). Full Janet
gate green (150 files). driver.janet is in the prelude fingerprint so the cache key
moved; prelude content is unchanged. jolt-chez fingerprint/ensure-prelude made
public for the test harness.
Ports the full # dispatch to the portable reader: #{} sets, #() anon-fns, #?/#?@
reader-conditionals, #_ discard, #' var-quote, #"" regex, #inst/#uuid/#tag tagged
literals, ## symbolic (Inf/-Inf/NaN), and #^ deprecated metadata. With this the
reader is feature-complete except position tracking + wire-in (inc 5d).
Reader-conditionals resolve clause-order against a portable feature set (atom
#{:jolt :default}); #? -> :skip / :form, #?@ -> :splice (the control protocol from
5b). #() uses the two-pass %-scan (collect indices, then rebuild replacing %N/%/%&
with gensym params) over the form tree via the jolt.host form-* contract. Three
host constructors added: form-make-set, form-make-tagged, form-gensym-name.
reader-parity 149/149. #() compares modulo gensym (canonicalize #-suffixed param
names by first-occurrence order — the two readers gensym different names but the
structure + %-mapping must match). ##NaN checked by the NaN!=NaN property. Full jpm
gate green (prelude pre-warmed). jolt-9ufe.
Ports list/vector/map literals and the quote family (' ` ~ ~@ @) + metadata (^)
to the portable Clojure reader. read-form now returns a [kind payload pos] control
triple (:form / :skip / :splice) instead of the Janet reader's :jolt/skip sentinel
FORMS — out-of-band control is collision-free and host-neutral (no tagged struct
to build or recognize). read-delimited dispatches the kinds; read-next-form skips
comments where a single datum is needed; read-map pairs k/v skipping trivia in
either slot. syntax-quote of a self-evaluating literal collapses at read time.
Four host constructors added to the contract (host_iface): form-make-list/vector/
map + form-sym-merge-meta (attach ^meta to a symbol). form-make-map reuses the
seed's reader-map (now public) for the source-order kv tracking. The portable
reader accumulates items in a jolt vector and the host builds its native form rep.
Gate: reader-parity 107/107 (lists/vectors/maps incl. nested + comments-in-coll,
quote/syntax-quote-collapse/unquote/deref, ^:dynamic/^Type/^{} meta). Full jpm gate
green (prelude cache pre-warmed — a cold cache races under the parallel gate when
the jolt-chez fingerprint changes; pre-existing, see new bead). jolt-sh1n.
fix-bugs-dont-reproduce, scoped per the keeper rule: jolt-if19 (a leading + on a
numeric literal errored instead of reading as the positive number) is fixed in
jolt.reader (read-number* now strips a leading + like -, positive), the code we
keep. The Janet seed reader (reader.janet) is left untouched — it's deleted in
Phase 5, so fixing it is wasted work.
Since the seed reader stays buggy, reader-parity can't use it as the oracle for
these inputs: added check-correct to assert the portable reader against the hand-
verified value (+5 => 5, +42, +0xff => 255, +3.5). reader-parity 67/67. No Janet
binary/gate impact (jolt.reader is not yet in the binary path). jolt-if19.
Starts taking the reader off Janet (src/jolt/reader.janet, 831 lines) into
portable jolt-core Clojure. jolt.reader holds the lexing/parsing LOGIC; form
construction + string->number parsing delegate to the jolt.host contract — a
Clojure source file can't write a {:jolt/type :symbol} literal (parses as a
tagged form) and the concrete representation is the host's to own. Same split the
analyzer/emitter already use. Once cross-compiled this runs on Chez so compile-
from-source needs no Janet reader.
inc 5a = the atom layer: whitespace/comments, symbols (+ nil/true/false),
keywords, strings (escapes), numbers (sign/hex/radix/ratio/fractional/exponent,
trailing N/M), characters. Collections, quote/deref/meta and dispatch (#) follow
in 5b/5c (throw not-yet-ported). Positions are char indices (Janet uses bytes);
identical for ASCII and the gate compares form VALUES, not positions.
host_iface.janet gains four reader primitives on the contract: form-make-symbol,
form-make-char, form-char-from-name, form-scan-number (the irreducible host bits
the portable reader rests on). Additive — new jolt.host interns, nothing else
changed.
Surfaced jolt-if19 (Janet seed reader: +N literals error instead of reading as N;
read-number strips only the - sign). The port reproduces it; both-throw counts as
faithful parity in the gate.
Gate: reader-parity 64/64 (symbols/keywords/strings/ints/hex/radix/ratio/floats/
exponent/N-M/chars). Full jpm gate green after clean rebuild, conformance 355x3.
jolt-50xx.
