init in compile mode is ~2.4 s (tier loading, analyzer self-compile, macro
recompilation), paid by every process that builds a ctx from source — each
jpm-test file, embedders, workers. init-cached marshals the built ctx to a
disk image (same root-env dicts as snapshot/fork) and later processes
unmarshal it in ~5 ms, any process: nothing from the baking process is
needed at load.
The cache key fingerprints the embedded .clj stdlib (which covers jolt-core:
analyzer, IR, core tiers), the .janet seed sources next to the module, the
janet version, the init opts, and the env knobs that shape a ctx (JOLT_PATH/
MUTABLE/AOT_CORE/FEATURES) — any change rebuilds. Corrupt or non-ctx images
fall back to a rebuild (unmarshal of garbage can 'succeed' with a scalar, so
the shape is checked, not just the throw). Writes are atomic (tmp + rename)
so racing cold starts never publish a torn image. JOLT_NO_IMAGE_CACHE=1
opts out; JOLT_IMAGE_CACHE_DIR overrides the location (default TMPDIR).
Test consumers switch to init-cached (harness, suite-worker, conformance,
the behavioral unit/integration tests); tests that validate the bootstrap
itself (bootstrap-fixpoint, staged-bootstrap, aot round-trip, direct-linking)
and the deps tests (tmp-dir :paths would fragment the key) keep real init.
Full jpm test: 2:46 -> 1:58 (~29%). New ctx-image-test covers cold/warm,
cross-process load (subprocess runs defn/redef/macros/protocols/multimethods
off the baked image), per-opts keying, and corrupt-image fallback.
The gate spent almost all its time rebuilding identical contexts: init is
~50 ms interpreted / ~900 ms compiled (tier loading, analyzer build, macro
recompilation), and both the conformance harness and the spec harness built a
fresh ctx PER CASE — 269 cases x 3 modes for conformance alone (~285 s), and
~1500 spec cases (~90 s). The suite battery additionally ran its 234 worker
subprocesses sequentially (~100 s incl. 5 x 6 s timeout files).
- api: snapshot/fork — marshal a fully-built ctx once (reverse-lookup dicts
from root-env, built at module load before any ctx exists), unmarshal cheap
fully-isolated deep copies (~2 ms). A fork shares nothing mutable with its
siblings, so per-case isolation is preserved exactly.
- conformance: one init per mode + fork per case (self-host pre-builds the
analyzer before snapshotting). 285 s -> 5 s, same 269x3 results.
- spec harness: jeval/run-spec/expect-throws fork from one lazy module-level
snapshot. Spec sweep ~90 s -> 9 s, all pass.
- clojure-test-suite: run the per-file worker subprocesses through a
token-channel worker pool (default 4, JOLT_SUITE_WORKERS to override,
capped at 8). 17.5 s with counts identical to the sequential run
(4046/520/116, 5 timeouts).
Full gate wall-clock: ~8 min -> 43 s, everything green (conformance 269x3,
fallback-zero, fixpoint, self-host, sci, staged, suite >= 4034/67, all
specs+unit).
adds self-hosted compiler is functionally:
- The default compile path is the portable pipeline using jolt.analyzer (Clojure) → host-neutral IR → backend.janet.
- The analyzer is itself Clojure, compiled by jolt for true self-hosting.
- bootstrap-fixpoint passes (stage1 == stage2 == stage3): rebuilding the compiler on its own output.
- clojure.core is now self-hosted in the overlay.
- Stateful forms (defmacro/ns/deftype/defmulti/require/in-ns) are interpreted by design.
(chan n xform) applies a transducer on the put side: a jolt transducer is a
directly-callable closure, composed over a reducing fn whose step gives each
output value into the channel (honoring its buffer kind). One put may yield zero
or more values; a reduced result (e.g. from take) closes the channel; close!
runs the transducer completion arity to flush stateful remainders. Works with
map/filter/mapcat/take/comp/etc.
Buffers: (buffer n) fixed, (dropping-buffer n) drops new values when full,
(sliding-buffer n) drops the oldest. Implemented via a non-blocking give —
(ev/select [ch v] closed-chan) detects a full buffer without parking.
harness: run-spec flushes per suite. spec: core.async/channel-transducers (5),
core.async/buffers (3). jpm test green.
Note: distinct/dedupe/partition-all/partition-by still lack a 0-coll transducer
arity in core (separate gap), so they can't yet be used as channel xforms.
Reader gaps caused the clojure-test-suite worker to crash whole deftests on
literals it could not parse (0N, 1.5M, 2r1010, 1/2), losing every assertion in
the file. read-number now handles:
- N (bigint) / M (bigdec) suffixes -> plain number (Jolt has no bignum/bigdec)
- ratios a/b -> double quotient
- radix integers NrDDD (2r1010, 16rFF, 36rZ) parsed by base
- exponents (1e3, 1.5e-2) and 0X hex
Also fixed suite measurement: when-var-exists now skips silently (its SKIP
print to stdout was corrupting the worker's count line, dropping whole files),
and the worker emits counts on an @@COUNTS sentinel line (robust against test
bodies that print, e.g. with-out-str). Runner parses the sentinel; deftest
crashes now report the underlying message.
Impact: clojure-test-suite 210->231 files run, pass 1955->3535, clean files
24->39. Baseline raised to 3450/38.
spec: numbers/literal-syntax (13 cases). jpm test green.
Run the external cross-dialect clojure-test-suite (lread/clojure-test-suite,
240 per-fn .cljc files in clojure.test format) against Jolt:
- test/support/clojure_test.clj: minimal clojure.test + portability shims
(deftest/is/testing/are + when-var-exists/thrown?/big-int?/lazy-seq?) — just
enough surface to load the suite and tally pass/fail/error. Pre-loaded so the
suite's (require [clojure.test ...]) finds it already populated.
- test/integration/suite-worker.janet: one-shot worker that loads the shim,
the suite's number-range helper ns, and a single .cljc file, then prints
'pass fail error'.
- test/integration/clojure-test-suite-test.janet: spawns a worker per file
under an ev/with-deadline wall-clock budget (so infinite-seq tests that hang
Jolt's eager evaluator are auto-contained, not a manual skip-list) and asserts
pass/clean-file counts stay at/above a baseline. References ~/src/clojure-test-suite
if present; skips cleanly when absent, like the jank battery.
Current: 210 files run, 7 timed out, 2233 assertions -> 1683 pass / 350 fail /
200 error, 23 clean files. Remaining fails are genuine divergences (float/ratio/
bigint, lenient transients where Clojure throws), tracked separately.
Fixes two real evaluator bugs the suite surfaced:
- :refer now preserves a referred macro's :macro flag (was interned as a plain
value, degrading referred macros to functions).
- resolve-var now resolves ns aliases (like resolve-sym), so aliased macros
(e.g. p/thrown? via :as p) dispatch as macros instead of being called as fns.
Reorganize the flat 49-file test/ into three layers (jpm test recurses, so all
are still discovered):
- test/unit/ white-box component tests (reader, evaluator, types,
persistent-map, lazy-seq, macro, interop, compiler)
- test/integration/ cross-cutting + regression batteries (conformance, jank,
sci-bootstrap/runtime, features, systematic-coverage, api,
core, namespaces, ported clojure suites) and
.../ports/ ported clojure/cljs test batches pending consolidation
- test/spec/ the behavioral contract (built out in following commits)
- test/support/harness.janet shared defspec table runner (cases compared via
Jolt's own =, with a :throws sentinel) + expect= helpers
Files moved with git mv (history preserved) and import paths fixed for depth.
jpm test green. README Test section updated.
Next: build out test/spec/ to cover the public API area-by-area, mining the
integration batteries and filling gaps.
