The eval path recorded only a frame's munged name, so a JOLT_TRACE backtrace was
a list of bare names. Register source for a runtime-compiled fn def when tracing
is on (keyed by the same munged name the entry push records), reusing the
source-registry the renderer already maps to "ns/name (file:line)". Direct-link
builds already registered via emit-def-cached; this covers the open-world eval
path. trace-off output is byte-identical (returns "" — seed mint / `jolt build`
unchanged), seed re-minted. A name shared across namespaces (e.g. -main) stays
bare, the existing ambiguity guard.
smoke asserts a file-backed project run maps a frame to ns/name (file:line).
The JOLT_TRACE opt-in was a top-level form in compile-eval.ss, so in a
self-contained joltc it ran at heap-build time — where JOLT_TRACE is always
unset — and never at runtime. `JOLT_TRACE=1 joltc -M:run` therefore produced no
trace from the distributed binary (it worked only under the source-loaded dev
launcher). REPL/nREPL tracing was unaffected (those enable at runtime).
Make it a jolt-trace-init-from-env! fn called from the runtime entrypoints — the
cli.ss dispatch and the built-joltc launcher — before any app namespace compiles,
so the app's own code is traced. While here, drop a redundant trace print in the
joltc launcher (jolt-report-throwable already emits it) that double-printed the
block once tracing actually produced one.
joltc-selfbuild-smoke asserts JOLT_TRACE=1 through the built binary yields exactly
one tail-frame trace.
A built binary loaded each namespace with (set-chez-ns! <ns>) and no restore, so
-main ran with *ns* left at the entry ns. clojure.main (and interpreted joltc)
run -main with *ns* = user, where a runtime (resolve 'alias/sym) is nil because
the alias lives in the entry ns, not user. Reset the current ns to user in the
launcher before -main so a compiled binary matches. build-smoke asserts it via a
separate two-namespace app (kept apart from the tree-shake app — a `resolve`
defeats tree-shaking).
Review turned up that the disable vocabulary was the exact lowercase strings
"0"/"false"/"no", so JOLT_TRACE=off (or FALSE, No, n) fell through and ENABLED
tracing — the opposite of intent — and the whole-run and dev-mode checks
disagreed on the empty string. Fold both into one jolt-trace-env-off? predicate
(case-insensitive, incl. off/n); empty/unset carries no signal (dev still traces,
a whole run still doesn't).
A repl or nREPL session now turns tail-frame tracing on, so an uncaught error in
evaluated/reloaded code shows a tail-frame backtrace with no JOLT_TRACE set. The
REPL and nREPL catch errors themselves rather than going through the uncaught
reporter, so they now print the history backtrace via a new jolt.host/backtrace-
string (history-only — the live continuation in a REPL is just REPL machinery).
Because the recording is baked in at compile time, only code compiled while a
session is live is traced; reload a namespace to trace already-loaded code.
JOLT_TRACE=1 still forces it on for a whole run (a plain -M:run traces its own
load); JOLT_TRACE=0 forces it off even in a session.
No seed change — jolt.main/jolt.nrepl are runtime-loaded and compile-eval.ss /
source-registry.ss are host files.
On the eval path nothing registers a source map, so jolt-backtrace-string
dropped every walkable frame and printed no trace at all. Keep any named,
non-plumbing continuation frame (rendered as a bare name when unmapped) so a
runtime error shows the surviving non-tail spine — "print what is available".
Add an opt-in tail-frame history behind JOLT_TRACE for the frames TCO erases.
Each compiled fn records itself on entry into a bounded ring-of-rings, MIT
Scheme's "history" shape: the outer ring holds one rib per non-tail subproblem,
each rib a small inner ring of the tail-calls made at that level. A tight tail
loop churns one rib instead of flushing the spine, so the non-tail caller
context survives and total space stays bounded. The reporter prefers this
history over the continuation when it's present, and resets it per top-level
form so an error's trace isn't padded with earlier REPL frames.
The emitter marks a tail call with (jolt-trace-mark! #t) so the runtime routes
the callee into the current rib vs a fresh one; a *tail?* dynamic var tracks
tail position (cleared by default, passed through if/do/let/loop/fn-body). It's
all gated on trace-frames?, which compile-eval turns on for JOLT_TRACE and
emit-image/`jolt build` force off — so non-trace emitted output is byte-identical
(prelude unchanged, seed re-minted), and a built binary carries no per-call cost.
jolt build -m ns on a namespace with only top-level side effects and no
-main produced a binary that printed its output, then crashed calling a nil
-main ("nil cannot be cast to IFn"). The launcher now calls -main only when
the entry ns actually defines one; otherwise the top-level forms (already run
at heap build) are the whole program and it exits cleanly. Regression case
added to build-smoke.sh.
Bring the language up to the 1.13.0-alpha1 changes that apply off the JVM:
- req! (CLJ-2949): a get-variant that throws "Expected key: k" on a missing
key, without nil-punning. The primitive behind checked destructuring.
- Checked-keys destructuring (CLJ-2961): :keys!/:syms!/:strs! bind and throw
when a key is absent; keys after & are declared-only (required for the !
variants, accepted otherwise) and create no binding.
- & is no longer a legal local binding in let/loop (CLJ-2954).
- Keyword-only array maps grow to 64 entries before going hash (was 8),
across the literal, assoc, and transient paths, so the common keyword map
keeps insertion order up to 64.
Skipped CLJ-2891 (JVM __init bytecode, JVM-only). 1.13 is still alpha, so
this tracks alpha1 and may shift. Regression tests in test/chez/unit.edn
(ahead of the JVM 1.12.5 the corpus certifies against). Seed re-minted.
jolt is Clojure, so a dep on org.clojure/clojure is always satisfied
intrinsically — the "skipping unsupported coordinate" warning on its
:mvn/version coordinate was just noise. Other unsupported mvn coords
still warn.
The nREPL server bound its loopback socket through libc process symbols,
which don't carry the socket entry points on Windows — they live in
ws2_32.dll and aren't in joltc.exe's export table, so --nrepl-server died
with "no entry for socket". Load ws2_32 before the foreign bindings there,
call WSAStartup once, and use Winsock's closesocket and int-typed
recv/send. Also fix SOL_SOCKET/SO_REUSEADDR, which the old macos-only
check got wrong on Windows.
Bake a version string into the self-contained binary at build time (from
$JOLT_VERSION, else git describe) and expose it via jolt.host/jolt-version:
--version / -V print it, and it shows in --help, the repl banner, and the
nREPL startup line. Dev runs off bin/joltc read it from git describe.
Add -e to the help output.
A user downloaded the auto-generated 'Source code' zip from the release
(no submodules) and hit the raw 'load failed for vendor/irregex/irregex.scm'.
cli.ss and make now check for vendor/irregex up front and print the fix
(clone --recurse-submodules / git submodule update --init --recursive);
README documents both and warns that GitHub's source archives can't build.
Release notes updated with the same pointer.
The -e path worked but jolt build died: jolt_petite_boot_len wasn't
foreign-entry-visible. -rdynamic's Windows equivalent for an exe is an
export table; -Wl,--export-all-symbols provides one.
A literal (foreign-procedure "chmod" ...) in compiled code becomes a fasl
relocation resolved when the boot loads — on Windows (no chmod/sigemptyset/
sigaddset in the CRT) that killed joltc.exe before any guard could run
(msvcrt abort, exit 3). jolt-foreign-proc-safe defers the lookup through
eval at evaluation time, where the guard works and a missing entry just
yields the fallback. chmod also skips the /bin/sh fallback on nt (execute
is by extension).
The built joltc.exe exited 3 (msvcrt abort) with no output on the -e smoke.
Link -static so the exe carries no libwinpthread/libgcc/lz4 DLL deps (needed
for distribution regardless), and add an ntldd + direct-run debug step.
Chez's system/process use cmd.exe on nt; every build command here is
written for sh. bld-sh-wrap spills the command to a temp script and runs
sh on it (no cmd quoting), identity on other platforms.
zuo's install target shells the unix installsh through cmd and dies. The
build tree already has everything: scheme.exe (boot files beside it, where
the Windows kernel looks), and scheme.h/libkernel.a/boots into a csv dir
that JOLT_CHEZ_CSV points the jolt build at.
The Windows PATH step embedded real newlines in its printf strings, which
broke the block scalar and invalidated the whole workflow file (the push
run failed at parse; workflow_dispatch refused). The wrappers are plain
echo pairs now.
rand-nth vec'd its argument, so (rand-nth nil) hit index-out-of-bounds
through the empty vector where the reference's (nth coll (rand-int (count
coll))) returns nil (and a set throws) — the last genuinely-fixable row in
the suite baseline; rand-nth is fully clean now. Corpus/README counts
updated to the current ~3570 rows, the run-corpus regression floor raised
from 2730 to 3390 to match current parity, and the stale traceability line
dropped.
A transient CDN timeout handed bash a 2-minute HTML error page instead of
linux-install.sh and failed the run. curl now fails on HTTP errors and
retries, and the script is sanity-checked before running.
Adds a windows-latest job to the release matrix: MSYS2/MinGW-w64 toolchain,
Chez 10.4.1 built from source (ta6nt), the same build-joltc flow, packaged
as a zip of joltc.exe. The whole job runs in the msys2 shell so cc/xxd/paths
behave; the produced binary is a plain Windows executable.
Platform seams: bld-nt? with the winsock/COM/registry link set, no -rdynamic
under MinGW, GetModuleFileName as the launcher's self-path on _WIN32, and
built binaries (joltc itself and jolt-build outputs) normalize to a .exe
suffix. workflow_dispatch added so the matrix can be dry-run without tagging;
the release upload step only fires on tags.
For #205.
Dialects without JVM interop can now filter the conformance corpus: :jvm
marks rows exercising host interop (java.*/clojure.lang.* class references,
dot forms, ctors, statics, arrays, proxy/bean), :common is portable Clojure
any dialect must satisfy. 3565 rows tagged (3175 common / 390 jvm), the key
documented in the row schema, and regen-corpus preserves it on rewrite.
Closes#289.
Auditing the remaining cts baseline for R7 exposed real contract gaps hiding
among the model residue — all fixed to reference behavior:
- stale no-ratio-era stubs: numerator/denominator now work over jolt's exact
rationals (non-ratio is the Ratio cast failure); rational? includes decimals
- casts and pending: peek/pop demand an IPersistentStack (pop nil is nil),
realized? demands an IPending (a plain list/range throws), transient demands
an editable COLLECTION (non-colls throw; the RFC 0003 sorted/list/seq
superset keeps the copy-on-write fallback), empty on a plain record throws
- nil and empties: (nth nil i) is nil, (nth nil i d) is d, a nil index is NPE,
keys/vals of anything empty are nil, (conj nil) is nil
- lookups: contains? on a string is index-only (other keys IAE), get on an
array is lenient (nth still throws), a VECTOR invocation has nth semantics
(([1 2] 5) throws — call position and jolt-invoke both)
- into only transients editable collections; a PersistentQueue/sorted target
folds through conj (RT's IEditableCollection split)
- numbers: number?/num accept BigDecimal, quot/rem throw on an Infinite/NaN
quotient, even?/odd? demand integers
- ordering: keywords compare namespace-first with nil first (Symbol.compareTo)
- misc: run! honors reduced, eval self-evaluates non-form values, intern
demands an existing namespace, counted? excludes strings, seqable? includes
arrays, shuffle rejects maps, sort-by rejects a collection comparator,
when-let demands one binding pair, case*/deftype*/letfn*/reify*/& are
special symbols
Two mis-certified corpus rows fixed (they threw on the JVM too and hid in the
tolerated bucket): a raw \d string escape and duplicate literal map keys.
SPEC.md gains the baseline-traceability section: every one of the 146
remaining suite failures maps to a documented divergence (integer-box,
no-single-float, RFC 0003 transients, seq/chunking model, stm-refs,
parse-uuid strictness, vec-array adoption). cts baseline 5955 -> 6042 pass,
5 errors, 30 namespaces. 9 JVM-certified corpus rows.
The reader now rejects what the JVM reader rejects: a token that starts
like a number but doesn't parse is NumberFormatException (1a, 08, 0x2g,
2r2 — never a symbol); ratio parts are digit runs (1/-1 invalid) with a
zero denominator throwing ArithmeticException; empty ns/name parts are
invalid tokens (:, ::, foo/, /foo) while /, ns//, and :/ stay valid;
duplicate map keys and set elements throw at read; unsupported string
escapes and octal escapes past \377 throw; a stray close delimiter is
'Unmatched delimiter'; \r ends line comments. #inst validates its
calendar fields progressively (leap years included) and #uuid demands
canonical hex. 1-arg symbol splits its ns at the FIRST slash
(Symbol.intern): (symbol "foo/bar/baz") is foo/"bar/baz".
clojure.edn gets its own strict seam (__read-form-edn): auto-resolved
keywords are invalid there, every #_ discarded form validates through the
same :readers/:default pipeline (an unreadable tagged element throws even
when discarded), built-in tags win over :default, M literals construct
BigDecimals, lists satisfy list?, and EOF honors :eof — an opts map
without :eof makes end-of-input an error.
clojure.edn-test.read-string goes 246 pass / 46 fail / 5 errors -> 297/0/0
(fully clean). cts baseline 5904 -> 5955 pass, 23 errors, 56 baselined
namespaces. 9 JVM-certified corpus rows; reader spec section.
The clojure.string case fns and searches now take any Object s through its
toString like the reference's ^CharSequence signatures ((upper-case :kw) is
":KW", (capitalize 1) is "1"); nil throws, and a nil substr in
starts-with?/ends-with? throws. some-fn re-ported with the reference
arities: (some-fn) is an arity error and a no-match result is the last
predicate's own falsy value (false, not nil). ifn? covers multimethods,
promises (which are now invocable — calling one delivers, via a cold-path
invoke-arm registry that costs the hot dispatch nothing), and deftypes
implementing IFn's invoke.
One structural find on the way: defmulti/defmethod deferred inside a fn
body (the deftest pattern) interned/resolved in whatever namespace was
current when they RAN, not the one they were written in — the macros now
bake their expansion ns and the setups honor it.
Also: Boolean/Integer/Double wrapper ctors, primitive TYPE statics
(Integer/TYPE etc.), .reduce on collections (IReduce), and Long/TYPE.
cts baseline 5857 -> 5904 pass, 58 -> 28 errors, 57 baselined namespaces —
the string cluster, some-fn, ifn-qmark, boolean-qmark, and reduce
namespaces are all fully clean. 7 JVM-certified corpus rows; spec entry.
byte/short/int/long/char silently wrapped or passed out-of-range values
through; the JVM range-checks (RT.byteCast family). One checked-cast
helper now carries the ranges: a double range-checks ITSELF before
truncating ((byte 1.1) is 1, (byte 127.000001) throws), NaN casts to 0,
ratios and bigdecs truncate, a non-number is CCE, and the throw carries
the JVM message. float range-checks against Float/MAX_VALUE. The
unchecked-* casts now genuinely wrap and sign-fold ((unchecked-byte 200)
is -56 — the old bit-and lost the sign) with doubles saturating like
Java's conversions; unchecked-long/int are host natives. double/float of
a bigdec convert instead of crashing. The no-single-float residue stays
accepted (SPEC.md).
Also fixes#290: a binary built by the SELF-CONTAINED joltc died with
'variable var-deref is not bound' when a namespace loaded at runtime.
The in-process build compiled flat.ss against a clean copy-environment,
which orphans every top-level define in locations the binary's runtime
eval can't see. It now compiles against the default interaction
environment (defines land in the real symbol cells, same as the legacy
fresh-Chez path) and a generated prologue pre-binds each kernel name the
runtime redefines to its kernel value, so the earliest boot reads match
the legacy path's primitive references. requiring-resolve is implemented
(the issue's dynamic-require pattern), and the release workflow smokes a
runtime require in a built binary.
Cast namespaces byte/short/int/long/char now fully clean; cts baseline
5805 -> 5857 pass, 67 baselined namespaces. 7 JVM-certified corpus rows.
add-watch/remove-watch/set-validator!/get-validator were atom-only; the
atom ctor ignored :meta and :validator; watching a var crashed. Now the
ARef contract is one seam: atoms keep their record slots (hot path
unchanged), every other reference type registers a predicate and stores
watches/validators in identity-keyed side tables, and notifies at its
mutation points. Vars notify on root changes (def on a watched var,
var-set outside a thread binding, alter-var-root — thread-binding sets
don't notify, like the JVM); agents notify per action. The def-var! wrap
costs two weak-table probes per def and does IRef work only on a watched
var.
Ctor options follow ARef: the validator gates the initial value
(IllegalStateException 'Invalid reference state' — also the class for
rejected swap!/reset!), :meta must be a map (else ClassCastException),
nil allowed. meta reads any reference through the identity side-table
(the type-gated fall-through is gone); alter-meta!/reset-meta! work on
non-var references.
Runtime-only (no re-mint). 9 JVM-certified corpus rows; spec entry; cts
baseline 5781 -> 5805 pass, 73 baselined namespaces (the residual error
in the watch namespaces is their STM ref section — refs stay out of
scope).
derive/underive/ancestors/descendants/parents/isa? re-ported from
clojure.core with the argument assertions and throw contracts intact:
derive asserts tag/parent shapes (AssertionError) and throws on redundant
or cyclic derivation; underive/derive on a non-hierarchy value throw at the
parents lookup (the map is called as a function, like the reference);
(descendants h SomeClass) throws UnsupportedOperationException. isa? gains
the reference's supers arm (a relationship derived on a class's super
applies to the class).
The class arms now answer fully through the one class graph: parents of a
class are its direct supers (bases), ancestors are the transitive set
rooted at java.lang.Object for concrete classes (interfaces are marked and
don't root at Object, matching getSuperclass semantics). deftype/defrecord
classes register into the graph at definition — protocol interfaces they
implement appear as supers (JVM-munged ns spelling), records carry the
record interfaces (IRecord/IPersistentMap/... whose closure supplies
Associative/Seqable), bare deftypes carry IType. The type NAME var still
holds the ctor (a jolt-ism); class-key maps it back to the class so
(ancestors TypeName)/(isa? x TypeName) work. canonical-host-tag learned to
NOT canonicalize deftype names through the graph arm (extend-type on a
deftype was registering under the bare segment its values never report).
Five old corpus rows used non-namespaced derive tags that throw on the JVM
too; now namespaced. 8 new JVM-certified corpus rows; spec entries for the
hierarchy family; cts baseline 5730 -> 5781 pass (ancestors/derive/
descendants/parents/underive namespaces fully clean), 74 baselined
namespaces.
Arithmetic and comparisons lowered to raw Chez ops, so an operand outside
Chez's tower (BigDecimal) crashed with a raw condition, and Chez contagion
leaked: (* 1.0 0) gave exact 0 where the JVM gives 0.0, (* ##Inf 0) gave 0
instead of ##NaN, (/ 1 0) raised an untyped error.
One seam now (host/chez/seq.ss): call position emits jolt-n* macros with the
both-Chez-numbers fast path open-coded; value position folds through the same
binary ops. Anything outside the tower falls to per-op slow hooks that
java/bigdec.ss extends, so bigdec arithmetic works in every position (the old
static-only :bigdec typing limitation is gone). JVM rules patched into the
fast path: a double operand wins, an exact zero divisor throws
ArithmeticException while a double zero divisor yields Inf/NaN, quot/rem/mod
cover ratios and doubles, min/max return the original operand with NaN
winning, a nil operand is NPE and a non-number CCE, zero-arg -// throw
ArityException at runtime instead of failing expansion.
Also: with-precision now binds *math-context* and bigdec results round with
real RoundingMode semantics (UNNECESSARY throws; division rounds to precision
instead of throwing); rationalize goes through the shortest decimal print
like BigDecimal.valueOf (the identity stub is gone); ratios coerce to bigdec
like Numbers.toBigDecimal; min/max int-literal operands no longer coerce to
flonum in the numeric pass.
Perf neutral: fib and seq benches unchanged (the fast path is two type checks
the optimizer folds); hinted fl/fx paths untouched. 19 JVM-certified corpus
rows; cts baseline 5614->5730 pass, 192->88 errors, 84->79 baselined
namespaces.
are let-bound its template vars, so a var inside quote never substituted:
(are [x] (special-symbol? 'x) if def) tested the literal symbol x twice.
Rebuild are on clojure.template/do-template (postwalk substitution), the
same architecture as upstream, with the same arg-count check.
This un-aborts every suite namespace whose are rows need substitution:
cts baseline moves 5302->5614 pass, 236->192 errors, 88->84 baselined
namespaces. The newly-reachable assertions also surface real divergences
now baselined and filed (edn reader strictness, Boolean ctor).
jank-lang/clojure-test-suite (per-core-fn clojure.test suites shared across
Clojure dialects) joins the default gate as vendor/clojure-test-suite, run by
host/chez/cts.sh: one joltc process per test namespace (a hang or crash is
contained by a per-process timeout), through the test/chez/cts-app project and
its cts-run runner, parallel workers.
Gating is exact per namespace against test/chez/cts-known-failures.txt, like
certify's allowlist: a namespace doing worse than the baseline fails, and one
doing better also fails as stale until the baseline is updated in the same
change. JOLT_CTS_WRITE_BASELINE=1 regenerates it; JOLT_CTS_NS runs a subset
verbosely.
Current standing: 243 namespaces, 5302 assertions pass, 340 fail + 236 error
across 88 namespaces pinned in the baseline (dominant clusters: BigDecimal
arithmetic operands, derive/ancestors hierarchy, transients, special-symbol?,
clojure.string case fns, the accepted narrow-int and seq-type-model
divergences). Two consecutive full runs produce identical counts. Wired into
make ci; skips cleanly when the submodule isn't checked out.
Five fixes shaken out by running jank-lang/clojure-test-suite:
- = short-circuits on identity like Util.equiv's k1 == k2, so (= s s) on an
infinite lazy seq answers true instead of walking forever. Numbers keep the
exactness-aware arm ((= ##NaN ##NaN) stays false like the JVM's).
- Calling a non-fn names the operator's CLASS in the ClassCastException, like
the JVM — never the value, whose printed form may be unbounded: ((range))
must throw, not hang rendering an infinite seq.
- realized? on a seq cell answers by its forced flag (the rest of a realized
lazy chain is a cseq under jolt's seq model), and the overlay's unsupported-
type error names the class, not the (possibly infinite) value.
- clojure.test/is dispatches a REGISTERED assert-expr method before its by-name
inline paths, like clojure.test where the built-ins are just pre-registered
methods — so an alias-qualified p/thrown? (the suite's portability helper)
isn't captured by the built-in thrown? path, which read its body as a class.
- clojure.test tracks tests and fixtures per namespace: deftest records its
defining ns, use-fixtures registers under the calling ns (no more cross-ns
clobbering), (run-tests 'ns ...) runs only those namespaces like clojure.test,
and each run-tests call prints/returns its own summary (global counters stay
cumulative for the n-pass/n-fail harness API).
Re-mint (20-coll.clj is seed; prelude only). +2 JVM-certified corpus rows;
the clojure-test fixture pins the alias-qualified assert-expr, per-call
summaries, and ns filtering.
The macos-13 Intel runner no longer gets allocated, so the x86_64-macos release
job queues forever. Ship prebuilt binaries for x86_64-linux and aarch64-macos;
Intel Macs build from source. The install script now says so instead of 404ing
on a missing asset.
next.jdbc's own source needs clojure.datafy + clojure.java.data and its tests
need JVM JDBC drivers, so it doesn't run on jolt. Keep clojure.jdbc (via
jolt-lang/db's jdbc.core over FFI SQLite) as the supported JDBC surface; point
migratus at jolt-lang/db.
The self-contained build reads the bundled Chez petite/scheme boots from the
joltc binary via foreign-entry on the embedded jolt_* symbols. On Linux dlsym
can't see an executable's symbols unless they're in the dynamic symbol table, so
'build' died with 'foreign-entry: no entry for jolt_petite_boot_len'. -rdynamic
exports them (macOS already resolves them). The new release self-contained-build
smoke caught this on the Linux runner.
`joltc build` inlines the runtime (host/chez/rt.ss and everything it loads, the
seed, compile-eval, loader, ffi, the vendored irregex) into each app binary by
reading those files off disk. That works from a jolt checkout but not from the
installed self-contained binary, which has no source tree:
joltc build -m app.core
=> Exception in call-with-input-file: failed for host/chez/rt.ss: no such file
build-joltc now bakes the exact transitive closure of files the build inlines
into the binary as embedded resources (keyed by the path the `(load "…")` forms
use), and build.ss/dce.ss read runtime source through bld-source-string, which
takes the embedded copy when present and falls back to disk otherwise. So the
same joltc builds apps both from a checkout and standalone.
The release workflow now smoke-tests a self-contained build (compile a tiny app
from an isolated dir, run it) — this is exactly what shipped broken, so it now
gates the release. buildsmoke/shakesmoke/staticnativesmoke unchanged and green.
Build tooling only — no re-mint, no runtime change.
install (root) downloads the self-contained joltc release asset for the host
platform, verifies its sha256, and drops the binary in /usr/local/bin (--dir /
--version override). Resolves the latest release via the GitHub API, clears the
macOS quarantine flag, and backs up an existing joltc. Modeled on babashka's
installer. README gets a one-line curl|bash install.
Document the register-class-supers! seam next to the other host-class hooks:
when a library class belongs to a hierarchy (a custom exception caught as
IOException, a value matching instance? across its supertypes and dispatching a
protocol extended to any of them), declare its supers once and instance?/isa?/
supers/extend-protocol all derive.
An unknown reader tag produced the reader's internal form
{:jolt/type :jolt/tagged :tag :#foo :form bar}, which tagged-literal? didn't
recognize and which leaked as a raw map when printed:
(tagged-literal? (read-string "#foo bar")) => false ; want true
(pr-str (quote [#foo bar])) => "[{:jolt/type :jolt/tagged ...}]"
Both the data path (rdr-construct-tag) and the compile path (emit-quoted) now
build a real tagged-literal for a tag with no registered reader, like Clojure's
*default-data-reader-fn*, so tagged-literal? / :tag / :form / printing all work.
clojure.edn reads raw forms through a separate __read-form-raw path and applies
:readers/:default itself, so it is unaffected.
Re-mint (backend + reader are seed sources); prelude byte-identical, image only.
make test green (selfhost holds, 0 new/stale), +2 unit rows.
jolt's own throw sites raised untyped Chez conditions with the class name buried
in an English message, so (class e) reported the opaque :object and only a broad
catch worked:
(class (try (Long/parseLong "xyz") (catch Throwable e e))) => :object
; JVM: java.lang.NumberFormatException
Raise typed throwables (jolt-host-throwable) at the Long/Double parse and
StringTokenizer sites so (class e) / .getMessage / a specific catch all reflect
the real class. And fold the exception supertype table (exception-parent) into
the one class graph: exception-isa? now resolves the simple name to its graph key
and asks jch-isa?, so exceptions and every other class share a single hierarchy.
Runtime only, no re-mint. make test green (0 new/stale), +2 corpus rows.
The "does value V declare method M; if so call it" decision was re-derived in a
dozen places with two different lookup helpers — records.ss jrec-cl and
collections.ss rec-coll-method were a byte-for-byte duplicate — and reduce only
honored a reify's own reduce method, not a deftype's:
(reduce + 100 (->Rng 5)) ; Rng deftype implementing IReduceInit
=> "not seqable" ; JVM: 110
Add iface-method / iface-call: one lookup that resolves a method for a deftype OR
a reify, with arity, and is the seam a core fn's interface arm collapses to.
jrec-cl now aliases rec-coll-method (the duplicate is gone). reduce routes its
IReduceInit arm through iface-method, so a deftype's reduce drives the reduction
like a reify's, reduced short-circuit included.
Runtime only, no re-mint. make test green (0 new/stale), +2 corpus rows.
record-method-dispatch was rebound with (set! record-method-dispatch ...) in six
files, each wrapping the previous binding, so precedence was whatever the rt.ss
load order happened to be — the true outermost arm was inst-time's Date arm, not
the one you'd guess. A type-gated wrapper that only whitelists its own methods
then errored on everything else, stealing universal Object methods from the arms
beneath it: (.getClass (java.util.Date.)) threw "No method getClass on Date",
same for File, while (class ...) and (.getClass "s") worked.
Replace the wrapper stack with an ordered list of arms (register-method-arm!,
ascending priority), each returning 'pass to defer. getClass is now one arm at
the top reached by every value, so it can't be shadowed; the three duplicate
getClass checks (dot-forms, host-static, base) collapse into it. Each former
wrapper is an arm at an explicit priority instead of an implicit load-order slot.
A library can register its own arm rather than set!-wrapping the dispatcher.
Runtime only, no re-mint. make test green (0 new/stale divergences), +1 corpus
row for getClass on Date/File.
instance?, extend-protocol dispatch, isa?/supers/ancestors, and the exception
hierarchy each read their own hand-kept table, and those tables had drifted:
(instance? clojure.lang.Associative [1 2]) was true but a protocol extended to
Associative wouldn't dispatch to a vector; keyword/IFn and seq/Seqable had the
same split; (isa? ExceptionInfo RuntimeException) was false and
(supers NumberFormatException) was empty.
Add one FQN -> direct-supers graph (class-hierarchy.ss) and derive the views
from it. value-host-tags builds on the graph closure so a vector reports
Associative/Indexed/ILookup/Counted/Seqable, a keyword reports IFn, a seq
reports Seqable/List/Counted, etc. instance? now tests membership in that same
list, so it can't disagree with dispatch. canonical-host-tag recognizes any
modeled class (was a separate literal set missing Seqable/ILookup/...).
class-direct-supers unions the graph edges and class-supers returns the
transitive closure, so the exception hierarchy answers isa?/supers/ancestors.
The graph is open: jolt.host/register-class-supers! lets a library graft its
own classes on and get every view for free.
Runtime only, no re-mint. make test green (0 new/stale divergences), +3
JVM-certified corpus rows.
joltc failed to start on Windows — "Exception in foreign-procedure: no entry for
pthread_sigmask". concurrency.ss resolves pthread_sigmask/sigemptyset/sigaddset at
load with a top-level (foreign-procedure …), which resolves its symbol eagerly;
those POSIX signal fns don't exist on Windows, so the whole runtime aborted.
Guard the three resolutions (like sched_yield/chmod already are) so a non-POSIX
host yields #f, and make jolt-set-sigint-blocked a no-op when they're unavailable.
The per-thread SIGINT mask is a POSIX-only optimization for the nREPL accept loop;
Windows delivers ^C through the console, and park-until-interrupt still parks on a
condition variable. macOS/Linux resolve the symbols as before — unchanged.
Running the whole rewrite-clj test suite (159 tests) surfaced seven more bugs;
with these it passes 3377/0/0. Each is a general jolt/JVM divergence:
- *out* was pinned to the startup stdout port, so (.write *out* …) escaped a
with-out-str capture (z/print writes via *out*). It now resolves the live
current-output-port, like print/__write, so a redirect is seen.
- nth / assoc past the end of a vector or seq threw a bare Chez error (class
:object). Throw IndexOutOfBoundsException, matching the JVM.
- A number's .toString(radix) ignored the base. Render in the base, lowercase
(rewrite-clj rebuilds 0xff / 0377 / 2r1001 through it).
- A required namespace's own :as aliases leaked into its requirer: the loaded ns
form compiles while (chez-current-ns) is still the requirer, so ce-scan-requires!
registered the loaded ns's aliases under the wrong ns and clobbered a same-named
alias there. Register an (ns NAME …) form's aliases under NAME.
- A quoted collection dropped its metadata; now it keeps USER metadata (drops the
reader's :line/:column/:file), like a Clojure quoted constant.
- enumeration-seq only did (seq e); it now drives a java.util.Enumeration through
hasMoreElements/nextElement, and StringTokenizer implements them.
Regressions: corpus rows (with-out-str/*out*, nth/assoc bounds, toString radix,
quote metadata, enumeration-seq) certified against JVM; a smoke fixture for the
alias leak (a required ns's alias must not leak). tools.reader + rewrite-clj added
to docs/libraries.md. make test green.
irregex builds a POSIX leftmost-longest DFA for a pattern when it can, and jolt
used it for everything. For a pattern with an alternation whose branches have
capturing groups, that DFA leaks a non-participating branch's group: e.g.
#"(?:([0-9])|([0-9])r([0-9]+))" on "2r11" left group 1 = "2" instead of nil, so
tools.reader (rewrite-clj's dep) misread 2r1100 as 2 and 16rFF as 16.
java.util.regex is itself a leftmost-first backtracking engine, so compile a
capturing pattern with irregex's backtracking matcher ('backtrack): its submatch
semantics match the JVM and it clears a losing branch's group. Non-capturing
patterns keep the DFA — with no groups to read, its whole-match result is all a
caller sees, and it avoids backtracking's worst case. The submatch count comes
from a first cheap compile; a capturing pattern recompiles once and caches.
This clears the last rewrite-clj parser-test failure (now 772/0/0). Corpus rows
for the alternation-group case and the radix read. make test green.
Running the rewrite-clj test suite under jolt exposed six bugs, each fixed here:
- `for`/`doseq` `:let` bindings never went through `destructure`, so a
destructuring pattern (`:let [{:keys [y]} x]`) hit `let*` raw and failed to
compile. Emit `let`, like Clojure.
- `with-open` couldn't close a deftype/defrecord that implements a `close` method
(java.io.Closeable / AutoCloseable, e.g. tools.reader's readers) — `__close`
only knew jhost readers and map `:close` fns. Dispatch a record's `close`.
- A deftype/defrecord method param named like a field didn't shadow the field
(the field's let-binding wrapped the params). Params now shadow, as in Clojure.
- A deftype whose simple name collided with a built-in host class clobbered it in
the global ctor table, so `(java.io.PushbackReader. …)` built tools.reader's
same-named deftype. Register deftypes/built-ins by FQN, don't let a deftype
overwrite a built-in's simple name, and qualify a bare `(Name. …)` to the
deftype's FQN only in the ns that defined it.
- `clojure.walk` was lazy over a non-list seq (missing `doall`), so a walk whose
fn has side effects read stale state. Make it eager, like Clojure.
- `Character/isWhitespace` used an ASCII-only check that missed U+2028 and other
Unicode whitespace. Use the JVM's Unicode set (minus the no-break spaces it
excludes).
Regressions: corpus rows (for-let destructure, method-param shadow, walk eager,
isWhitespace), a unit row (with-open closes a record), and smoke checks (the
class-name collision, run in a fresh -e process so the deftype doesn't leak).
One divergence remains unfixed: a submatch from a losing regex alternation branch
leaks when the winning branch has a quantified group (a bug in the vendored
irregex engine, not jolt) — tracked separately.
A registered #tag data reader whose fn returns a FORM (borkdude/html's #html
expands to (->Html (str …))) was rewritten to a runtime call (reader-fn 'inner),
so the returned code became a runtime list value instead of being compiled —
(str #html [:div]) rendered the code, not "<div>". Clojure applies a data reader
at read time and substitutes its result as code.
loader.ss now applies the reader at load time: a code form (a list) is spliced in
to be compiled, a value (time-literals #time/date -> a Date) keeps the runtime
call, which also keeps a non-serializable constant out of an AOT build. The build
emit path never applied data readers at all (a #tag literal failed a `jolt build`
with "unsupported form"); emit-image.ss gets an ei-emit-form-hook the build sets
to the same rewrite, left as a no-op elsewhere so the seed mint (which doesn't
load loader.ss) is unaffected and the self-host byte-fixpoint holds.
Also make clojure.test report the actual values of a failing (is (= a b)) — it
printed only the form. Restricted to the common pure predicates so a macro head
still takes the plain path.
Fixture test/chez/datareader-app + a smoke check (interpreted) and a build-smoke
check (AOT). make test green, no corpus change.
A :jolt/native spec can now carry a :static archive; `jolt build` links it
into the executable, so the app calls the C code with no shared object on the
target. --dynamic (or :jolt/build {:dynamic-natives true}) keeps the old
runtime load-shared-object behavior; a spec with no :static is unchanged.
The cc link force-loads the archive (-force_load on macOS, --whole-archive on
Linux) and exports the executable's symbols (-rdynamic on Linux) so the baked-in
symbols resolve via (load-shared-object #f) + foreign-procedure at startup. Build
step 1 evaluates the app's foreign-procedure forms in-process, so a static
archive is preloaded there as a throwaway shared object to resolve them.
The distributed self-contained joltc has no external cc/Chez but must build these
apps, so it now bundles the Chez kernel (libkernel.a + scheme.h) and the launcher
source and re-links a custom stub with the archives baked in — needing only a
system cc, no Chez. run/repl skip static-only specs (nothing to load); keep a
:darwin/:linux candidate to use such a lib interpreted.
Adds static-native-smoke (cc path) to ci and a static phase to the joltc
self-build smoke (distributed path).
repl-form-complete? entered the :regex state on '#' but only consumed the
'#', so the opening '"' was then read by the :regex handler as the CLOSING
quote. The regex body got scanned in :code state, and any delimiter or quote
inside it (a group like #"(a)", a char class #"[0-9]+") threw off the
paren/string count — so a one-line regex form was judged incomplete and the
REPL hung waiting for continuation lines. Consume the '#"' together.
Adds a self-checking predicate test (test/chez/repl-reader-test.clj, run via
joltc so jolt.main resolves) and an end-to-end regex REPL case in smoke.sh.
The :expected was a bare list "(0 :a 2)", but run-corpus evals :expected as
source, so it applied 0 as a fn -> "0 cannot be cast to IFn". Every other
list-valued :expected is self-evaluating; this one slipped in unquoted.
Vectorize it to [0 :a 2], matching what regen-corpus.clj produces.
The REPL evaluated one line at a time, so a form split across lines
(e.g. `(+` then `1 2)`) raised instead of waiting. The read loop now
accumulates lines until delimiters are balanced — skipping string,
char, regex and comment context — printing a `... ` continuation prompt
for each extra line.
Reader/runtime errors rendered as Chez's "attempt to apply
non-procedure #[chez-pmap...]" instead of their real message. Two causes:
jolt-throw raised the thrown value raw. When a throw crossed the host
`eval` boundary, Chez re-wrapped the non-condition into a compound
condition whose message extraction applies the value, losing the message
and crashing on ex-info's empty-map :data. jolt-throw now raises a
&jolt-throw condition wrapping the value; catch (lowered to `guard`),
jolt-report-uncaught and jolt-render-throwable unwrap it back via
jolt-unwrap-throw, so ex-data/ex-message and the backtrace tag survive.
Every reader/post-prelude EOF-throw site used `(empty-pmap)` (with
parens), applying the empty-map value as a procedure and crashing during
ex-info construction before jolt-throw ran. Fixed to `empty-pmap`.
Re-minted the seed; smoke 23/23, unit 574/574.
^C to a running `joltc --nrepl-server` aborted with "thread does not
own mutex" because the accept-loop thread absorbed SIGINT in its foreign
accept() call, where Chez can't run the keyboard-interrupt handler, and
run-main-pump's tight condition-wait loop wasn't interruptible anyway.
Block SIGINT in the primordial thread before starting the server so the
accept loop inherits a blocked mask, park in a single interruptible
condition-wait via the new park-until-interrupt, and run registered
shutdown hooks (newest-first, each isolated) from the keyboard-interrupt
handler before (exit 0). The stop fn now drops .nrepl-port via the new
jolt.host/delete-file seam — clojure.java.io/delete-file doesn't exist
in Jolt and silently no-ops, so .nrepl-port was never removed.
^D (EOF) exits cleanly in canonical mode but some terminals and editors
don't deliver it, leaving the user stuck. Accepting :repl/quit or :exit
as the first form of a line gives a reliable keyword exit that works
everywhere. The check parses the line with read-string rather than
checking the evaluated value, so a nested value that happens to print
as the keyword can't trigger an exit.
list?, ratio?, and rational? are the predicate-web members that are
genuinely safe to migrate: not extended at runtime, not on the compiler
emit/inference path, not reached by the kernel tier. They now live in the
overlay (clojure/core/20-coll.clj) built on the jolt.host tower/rep tests,
lowering to the same code the native shims did. Removed their native
definitions (predicates.ss) and, for ratio?/rational?, the now-redundant
post-prelude re-assertions. Also dropped the dead all-flonum overlay
ratio?/rational?/decimal? stubs.
The rest of the web stays native and is documented as such: map?/set?/
seq?/coll? are extended with sorted/record/lazy arms, decimal? is extended
by the optional bigdec module, integer?/float? are on the emit/inference
path, vector? is reached by the kernel-tier peek. jolt.host exposure is
therefore narrowed to just the tests these three consume (exact?,
rational-type?, cseq?, cseq-list?, empty-list?).
Numeric probe is byte-identical to pre-migration; list? correct across
list/vector/lazy/empty/cons/rest cases. Selfhost fixpoint holds, values/
unit/smoke/corpus green, bench flat within noise.
The clojure.core type predicates bottom out at host tests that overlay
Clojure can't reach. Expose them under jolt.host so the predicate web can
be built as pure compositions that lower to exactly these calls:
numeric tower: exact? flonum? integer-type? rational-type?
collection reps: pvec? pmap? pset? cseq? empty-list? cseq-list? lazyseq?
exact? is wrapped to be total (Chez's raw exact? errors on a non-number;
the others return #f for a non-match). lazyseq? is exposed in
lazy-bridge.ss because jolt-lazyseq? is defined there, after predicates.ss.
map?/set?/seq? are deliberately not reduced to a single rep test: they are
extended at runtime with sorted-collection/record/lazy arms, so only the
rep predicates are exposed, not those unions. Additive only (new bindings,
nothing references them yet); bench unchanged within noise.
Attempting to migrate the reader-conditional constructor to the overlay
revealed that an overlay defn returning a :jolt/type-tagged map literal
silently fails to bind during the seed mint: the guard around each
prelude form swallows the load-time error, leaving the var unbound. This
is the same reason every other tagged-value constructor (atom,
volatile!, tagged-literal) is native, so reader-conditional is
reclassified STAY-PRIMITIVE rather than a safe migration.
set was a native shim (apply jolt-hash-set (seq->list coll)). It is a
pure composition, so the Clojure version (apply hash-set (seq coll))
lowers to the same code. The compiler uses set, but only off the emit
path (the backend's bare-native-names def and type inference), so it can
live in the kernel tier: compiling that tier never calls set, and by the
time those callers run the tier is already bound.
This is distinct from boolean, which the backend calls for every :if
node on the emit path. Moving boolean even to the kernel tier deadlocks
(compiling the tier that defines boolean needs boolean), so boolean stays
native. Added a comment in predicates.ss recording that.
Re-mint converges in 3 passes and the benchmark suite is unchanged
within noise (collections 43.3 vs 43.1, binary-trees 367 vs 367, the
rest flat).
The overlay defines transduce in clojure/core/22-coll.clj as a pure
composition (xf (reduce xf init coll)), and it shadows the native
jolt-transduce by load order. The compiled overlay version is already
what gets baked into the seed, so the native binding in
natives-transduce.ss was dead weight.
transduce is not used by the self-hosted compiler and no overlay tier
before 22-coll references it, so removing the native binding is safe.
Re-minting produces a byte-identical seed, which proves the runtime is
unchanged. sequence stays native (its transformer iterator drives the
reduced box and lazy realization directly).
range, map, and filter were fully element-by-element lazy, so
(map f (range 1 50)) realized one element per first/nth where JVM
Clojure realizes a whole 32-element chunk. range is a chunked
LongRange on the JVM and map/filter are chunk-preserving, so the
observable side-effect timing differed.
Following clojure.lang.LongRange, ChunkedCons, ChunkBuffer and
core.clj, this adds a crest field to the cseq record and a
cseq-chunked constructor modeling ChunkedCons (a standalone chunk
pvec, an offset, and the after-chunk seq). The chunk accessors move
to seq.ss next to the representation they read. map/filter/remove
take a chunked branch when the source is chunked, realizing the whole
chunk and chunk-cons'ing it onto a lazy rest, so their output is
itself chunked and chained transforms each batch by 32. Bounded range
is now an eager chunked seq, and the reduce fast path flows through a
ChunkedCons rest. The chunk-buffer/chunk/chunk-cons builder API in
natives-array.ss now produces a real ChunkedCons.
Single-arg (range), multi-coll map, and plain lazy seqs stay
element-by-element, like the JVM.
Adds a lazy / chunking suite to the corpus that observes realization
timing via an atom counter: first over a chunked map realizes 32,
crossing a chunk boundary realizes 49, chained maps batch [32 32],
filter applies the predicate to the whole first block, and a plain
lazy seq still realizes one element at a time. Two cases that
documented the old over-laziness now assert the JVM value of 32 and
were dropped from the allowlist. certify against JVM Clojure 1.12.3
reports 0 new and 0 stale divergences.
On a pushed v* tag, build the self-contained joltc (make joltc-release) for
x86_64-linux, x86_64-macos, and aarch64-macos, package each as a tar.gz plus a
SHA256, and attach them to the GitHub Release. Linux builds Chez from source like
tests.yml (the apt package lacks the kernel dev files build-joltc cc-links
against); macOS uses Homebrew chezscheme, which ships chez and the csv kernel
files. No notarization, matching dirge — macOS tarball users de-quarantine once
or install via a Homebrew tap.
The Homebrew tap update job is a separate follow-on; this covers building and
publishing the release assets.
host/chez/build-joltc.ss builds joltc into target/<profile>/joltc: it emits a
flat source of the full runtime + compiler image + inlined build.ss + every
jolt-core/stdlib file as a baked string literal + a cli.ss-style launcher, then
(in a fresh Chez, so the inlined runtime's redefinition of error doesn't strand
early references and runtime eval still sees the runtime's top-level procedures)
compiles it and cc-links it with the Chez petite/scheme boots and the launcher
stub embedded as C arrays. The launcher reads those arrays via FFI on
(jolt-materialize-bundles!) and registers them so build-self-contained can spill
them. joltc itself is cc-linked (clean signature for Homebrew); only the apps it
later builds use the appended-stub path.
build.ss: skip the csv toolchain check on the self-contained path and create the
build dir with a subprocess-free bld-mkdir-p, so a from the
distributed binary shells out to nothing.
release = optimize-level 3 + no inspector info + compressed; debug =
optimize-level 0 + inspector + procedure source + debug-on-exception.
joltc-selfbuild-smoke.sh (make joltcsmoke) builds joltc and, with an empty
environment (no chez/cc/PATH), drives it through the build-app fixture, asserting
the produced binary's output. .gitignore ignores target/.
Adds the pieces a toolchain-free joltc needs to compile apps with no external
Chez or cc:
- host/chez/java/io.ss: register-embedded-bytes!/jolt-embedded-bytes,
read-file-bytes, jolt-spill-embedded!, jolt-append-payload! (frames an app
boot onto the stub as [stub][boot][len:le64]["JOLTBOOT"]), and jolt-chmod-755
via load-shared-object #f (no subprocess).
- host/chez/stub/launcher.c: a native stub that locates its own executable,
reads the trailing frame, and hands the appended boot to the Chez kernel.
- host/chez/loader.ss: resolve-on-roots consults the embedded source store before
disk; ldr-read-source reads baked source. Dev (empty store) is unaffected.
- host/chez/build.ss: build-binary step 4 splits into build-self-contained
(in-process compile-file/make-boot-file with the system error restored, then
append the boot to a copy of the embedded stub) and build-with-cc (the existing
dev path). The self-contained path is taken only when the stub is embedded.
The legacy cc path is unchanged behaviorally; make buildsmoke still passes.
The overlay comment claimed Jolt has no chunked seqs and that chunked-seq?
is always false. That is no longer true: a vector's seq is a real chunked
seq, and post-prelude.ss rebinds chunked-seq? to na-chunked-seq?, which
returns true for a vector seq. The defn here is only a placeholder so
references compile during overlay load. Updated the comment to say so.
Two pre-existing issues in the nrepl command, exposed when #269 moved the accept
loop into a future.
jolt.nrepl/start was invoked inside (future ...), so binding the socket happened
on a background thread. A bind failure such as the port already being in use was
captured into a future that nothing derefs and silently swallowed, leaving the
main thread parked in run-main-pump forever with no server and no error. start
now binds the socket synchronously before returning, so that failure propagates
to the caller and the process exits with a visible message. Only the blocking
accept loop runs on a worker thread.
There was also no shutdown path: the accept loop ran forever and the listen
socket was never closed. start now returns a stop fn that breaks the accept
loop, closes the socket to free the port, and removes .nrepl-port. main.clj runs
run-main-pump and then calls the stop fn, so a stop-main-pump (from a glimmer app
on quit, or from nrepl-evaluated code) shuts the server down cleanly and the
process exits.
PR #269 added the main-thread executor (call-on-main-thread, run-main-pump,
stop-main-pump) so the nREPL accept loop can run on a worker thread while the
primordial thread owns the GUI main loop. Two problems made stop-main-pump
unusable as a graceful-shutdown or external API.
run-main-pump set jolt-main-pump-active to #t on entry but never cleared it on
exit, so after the pump returned the flag stayed #t. call-on-main-thread also
read that flag outside the queue mutex, so even with a reset there was a window
where a job could be enqueued just as the pump left, then block forever on a
pump that was gone.
Both are now decided under jolt-main-queue-mu. The pump clears active in the
same critical section where it sees the stop flag and an empty queue, and
call-on-main-thread reads active and enqueues atomically under that lock. A
caller that loses the race sees the pump inactive and runs the thunk inline,
the same fallback used when no pump is running, rather than blocking. A
dynamic-wind around the loop also clears active on an abnormal exit so a later
run-main-pump starts clean.
Match babashka's spelling: the nREPL server now starts with
`bin/joltc --nrepl-server [port]` instead of `bin/joltc nrepl`. Port
parsing and JOLT_NREPL_PORT are unchanged.
Also wire up --help/-h to print usage (previously only the no-arg
invocation did), and fix the usage listing to show the real flag.
Smoke now asserts --help mentions --nrepl-server. Docs updated to match.
The AOT suite doesn't cover 64-bit integer arithmetic (Chez fixnums are 61-bit,
so genuine 64-bit values are bignums) — the SplitMix PRNG behind test.check is
the worst case. Add the measured jolt-vs-JVM numbers for the PRNG/mix-64 and the
generator workload: the bitwise native-ops + var-cell caching took mix-64 from
~18x to ~3.2x JVM and the PRNG from ~30x to ~12x; the residual is the open-world
generator dispatch/allocation and the bignum floor, not arithmetic.
Profiling jolt-i5if showed <=60-bit arithmetic is already native-fast; the real
general overhead in the run/-e/-m path is var resolution. Every var reference
compiled to (var-deref ns name), which builds + hashes a fresh "ns/name" string
and does a hashtable lookup per access (~45ns). The var cell is interned and
def-var! mutates it in place, so caching the resolved cell is sound under
redefinition.
Generalize the devirt per-site cache-cell mechanism to var value references: a
ref inside a fn resolves its cell once into the def's closure, then reads it via
var-cell-deref (a field read after the first). var-cell-deref is the cell-based
var-deref — binding-aware (dynamic vars + *ns* still resolve) and lenient on an
unbound root (a forward-declared var doesn't throw, unlike jolt-var-get).
Gated by a runtime flag: ON for runtime-compiled code (compile-eval.ss), OFF for
the seed mint and AOT build (emit-image.ss) so the seed stays a byte-fixpoint --
prelude.ss is unchanged, only image.ss picks up the new backend. ~5x on a
var-ref-heavy loop (1058ms->205ms); ~1.2x on test.check (its generators are more
deftype/dispatch-bound than var-deref-bound). No C/FFI.
Corpus rows pin redefinition / dynamic binding / forward ref through a cached
ref. make test + shakesmoke green, selfhost holds, SCI 211/218, certify 0-new.
Profiling the test.check distribution/large-sample slowness (jolt-i5if): the
hot path is the SplitMix PRNG, dominated by 64-bit mix arithmetic, and the
bitwise ops (bit-and/or/xor/not, shifts) were NOT in the backend native-ops
table — so (bit-xor a b) compiled to a var-deref through the variadic overlay
(__bit-xor) instead of a direct call, the way +/-/* already emit.
Map bit-and/or/xor/not to the Chez bitwise-and/ior/xor/not primitives (inlined
to native code; a non-integer operand now errors like the JVM instead of being
silently truncated) and the shifts to a direct helper call. bit-and-not stays on
its overlay — its only Scheme impl is 2-arg, so a value-position arity-3 use
would mis-emit.
mix-64 arithmetic 2.7x faster, raw split+rand-long 2.4x, gen/vector ~1.4x. The
remaining gap is the bignum-vs-native-long floor (~20x, substrate) plus the
generator machinery (deftype/fn dispatch, separate). Corpus rows added for value
position, bit-not, apply, and a full-64-bit unsigned shift.
jolt's `is` was a fixed macro with no assert-expr multimethod, and the runner
bypassed the report multimethod, so libraries couldn't register custom
assertions or custom report types (e.g. test.check's ::trial/::shrunk).
Add assert-expr (2-arg [msg form], dispatch on the form's first symbol /
:default / :always-fail), do-report routing through report, and report
:pass/:fail/:error methods that feed the counters. `is` dispatches to an
explicitly-registered assert-expr method before its inline path, so thrown?/
thrown-with-msg?/= and every built-in form stay byte-identical.
Runtime stdlib only, no re-mint. test/chez/clojure-test.clj self-checks the
extension points + full is/are/testing/thrown?/use-fixtures surface; smoke gate
runs it.
jolt unifies every integer as one exact-integer type, so (byte/short/int n)
report Long not Byte/Short/Integer and instance? Byte is false. Confirmed
substrate-inherent: (byte 5) is a Chez immediate identical? to 5 (nothing to
tag, numbers carry no metadata), and arithmetic compiles to a raw Chez + that a
boxed narrow type would crash. Value/arithmetic/equality are correct.
Certify the value-correctness (= to plain int, arithmetic promotes, is a Number)
and pin the class/instance? divergence under a new :integer-box-model category.
Data/doc only.
A 62-case jolt-vs-JVM probe across seq type identity, chunking
granularity, eagerness, and realization timing. Findings: the whole
producer family is lazy at construction (no eager bugs remain), and the
26 divergences fall into two classes that diverge by representation, not
value.
Lock in the laziness contract as certified corpus rows: construction=0
for keep/keep-indexed/map-indexed/distinct/partition-by/partition-all/
interpose/interleave/take-nth/reductions/tree-seq/replace, sequence
realizes 1, next realizes 2, rest realizes 1.
Pin the two accepted divergence classes (allowlisted, gate-guarded):
- seq-type-model: jolt reifies seqs as PersistentList/LazySeq vs JVM's
Cons/Iterate/LongRange/Repeat/Cycle/ChunkedSeq/StringSeq/KeySeq/RSeq/
ArraySeq/SubVector (jolt-aei7)
- chunking-model: unchunked, realizes one where JVM realizes a 32-chunk;
mapcat/dedupe fully lazy at construction (jolt-mm6v)
known-divergences.edn gains both categories; SPEC.md documents the seq
semantics contract. Data/doc only, no re-mint. certify 0 new / 0 stale.
A pvec is a 32-way trie, but na-chunk-first built each block by calling
pvec-v on the full backing vector — materializing all n elements to a
flat Scheme vector — then copying the 32-wide window out of it. That made
chunk-first O(n), so walking a vector chunk-by-chunk (Clojure's real
chunked map/filter fast path) was O(n^2): a ported chunked map over 500K
elements took 39s, superlinear to ~700s at 2M.
na-chunk-size equals pv-width and blocks are 32-aligned, so a block is
exactly one trie leaf — pv-chunk-for hands it back in O(log n). Copy that
leaf directly; fall back to per-index reads for the rare window that
crosses a leaf boundary. Chunked map is now linear, ~133x faster at 500K
(293ms) and within ~2.3x of the native seq loop, which makes a
clojure-in-clojure seq tier viable.
Corpus rows pin chunk-first window contents + chunk-rest boundaries
against JVM; fixed a stale 'always false' chunked-seq? label.
The corpus compares values, so eager-vs-lazy was invisible (identical
values). Add rows that reduce laziness to a value via a side-effect
counter: realization order (map/filter left-to-right), exact realization
count under take/nth/drop (no over-realization), and lazy-seq
memoization (realize-once across repeated walks). Sourced through
unchunked producers (iterate, lists) so jolt's unchunked model matches
the JVM. All certify against Clojure 1.12.5.
map/filter/remove/take/drop/concat/take-while/drop-while/mapcat/partition
built an eager-headed cseq: the first element (and the fn application) ran
at construction, so a side-effecting (map f coll) fired f immediately and
(class (map …)) was PersistentList instead of LazySeq. This diverged from
Clojure, which wraps the whole body in lazy-seq. It went unnoticed because
the conformance gate certifies values, not realization — eager and lazy
heads produce identical values — and unit.edn even baked PersistentList in
as expected. test.check's for-all-takes-multiple-expressions (which counts
side effects in a for-all body) exposed it.
Wrap each native producer's result in a lazy-seq node so the body, incl.
the first element, defers until forced — the forced cseq still has eager
heads, so reduce/count/dorun/etc. force on walk and there's no per-element
cost. dedupe's (seq coll) is moved inside its lazy-seq. A jolt LazySeq is
now recognized by coll?/empty, the analyzer's form predicates (a macro can
build its expansion with map), value-host-tags + instance? (LazySeq/ISeq/
Sequential), and reports clojure.lang.LazySeq.
Kept the native Scheme implementations rather than porting Clojure's: a
straight lazy-seq+cons port is 3x slower and Clojure's chunked fast path is
288x slower because jolt's chunk machinery is unoptimized (filed jolt-j9dz);
the wrapped natives are Clojure-lazy at native speed.
+12 corpus rows (laziness at construction, LazySeq type, both JVM-certified).
make test + shakesmoke green, selfhost holds, 0 new divergences.
Chez fixnums are 61-bit, so the bignum bitwise-and mask allocates for any
value past 2^60 — and unchecked-* ran it on every result, even small
in-range ones. An exact integer already in [-2^63, 2^63) is its own wrap,
so return it directly; only an out-of-range result (a multiply overflowing
into 128 bits) needs the mask. ~30% on in-range unchecked-add loops,
neutral on full-64-bit multiply.
Note: the 64-bit arithmetic floor on Chez stays ~31ns/multiply (bignum, no
native 64-bit int); the test.check distribution hangs are dominated by
generator/dispatch overhead, not arithmetic — this is a general win for
long-heavy code, not a fix for those.
jolt-unc{add,sub,mul,inc,dec,neg}2 wrapped every non-flonum result to a
signed 64-bit integer, so (unchecked-add 2/3 2/3) truncated to 1 instead
of 4/3. Under *unchecked-math* the analyzer rewrites +/-/* to unchecked-*,
so any ratio arithmetic in such a file silently floored. Clojure's
unchecked-add falls back to regular arithmetic for non-primitives; only
long math wraps. Wrap iff both operands are exact integers.
Shaken out by test.check's gen/ratio monoid property (the + and 0 monoid
held for small-integers but failed for ratios).
Close clojure.spec.alpha's remaining gaps — its conform/explain/describe/multi-spec
suite (clojure.test-clojure.spec, multi-spec) now passes fully.
- (get reify k) / (:k reify) routes to a reify's clojure.lang.ILookup valAt. spec
reifies fspec/regex specs as ILookup and reads (:args spec) off them, so before
this instrument never saw the args spec.
- A failed numeric comparison reports the JVM class: a nil operand is
NullPointerException, a non-number is ClassCastException (was an opaque :object
condition). conform-explain checks the thrown class.
- A quoted / macro-form #inst / #uuid literal constructs its Date/UUID value, like
the JVM reader (which builds it at read time). emit-quoted was emitting the raw
tagged form, so #inst "1939" and #inst "1939-01-01T00:00:00.000-00:00" weren't =.
- An anonymous fn reports class clojure.lang.AFunction$fn (the $fn marker), so
spec's fn-sym returns ::s/unknown for it, matching the JVM's ns$fn__N.
- A fn with & {:as m} kwargs accepts a trailing map (Clojure 1.11): (f :a 1 {:b 2})
and (f {:a 1}) both bind m, by merging an odd trailing map over the pairs.
- A thread responds to .getStackTrace (empty — jolt does TCO).
clojure.test-clojure.instr does not fully pass: its ::caller assertions need the
calling fn's stack frame, which TCO erases (an inherent host divergence, like the
JVM keeping tail frames).
make test green (+4 corpus rows, 0 new divergences), shakesmoke byte-identical.
Re-mint (backend emit-quoted + the destructure macro).
jolt's reader had no case for #: , so #:event{:type :search} died as an unknown
tagged literal. Now #:ns{...} qualifies each bare keyword/symbol key with ns
(:_/x stays unqualified, an already-qualified key is left alone); #::{...} uses
the current ns and #::alias{...} resolves the alias — matching Clojure.
clojure.spec.alpha's multi-spec test (which builds #:event{...} event maps) now
passes.
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (the reader is a seed source).
jolt fns reported (class f) = clojure.lang.IFn, so they carried no defining
symbol — clojure.spec.alpha's fn-sym (which reads a fn's class name to recover its
symbol) produced garbage, so explain-data's :pred for a bare-fn predicate was `/`
instead of e.g. clojure.core/keyword?.
Now def-var! records proc -> (ns . name) (first def of a proc wins, so an alias
like (def inc' inc) doesn't rename inc), and jolt-class-name returns "ns$munged"
for a known fn — matching the JVM, where (class odd?) is clojure.core$odd_QMARK_.
A munged fn class's ancestors include clojure.lang.AFunction's hierarchy
(IFn/AFn/Fn/Runnable/Callable), so (ancestors (class f)) still holds. Anonymous /
unregistered fns stay clojure.lang.IFn (fn-sym yields :unknown, as on the JVM).
This fixes explain-data / s/form / s/describe of bare-fn predicates in
clojure.spec.alpha (and unblocks parts of its suite + test.check's reporter test).
make test green (+1 corpus row, the (type inc) unit row updated to the JVM value),
shakesmoke byte-identical, runtime only (no re-mint).
General fixes from clojure.spec.alpha's test suite.
- (symbol a-var) returns the var's qualified symbol (clojure.spec.alpha/->sym).
- clojure.lang.Compiler/demunge reverses Clojure's name munging
("clojure.core$odd_QMARK_" -> clojure.core/odd?); spec's fn-sym uses it.
- clojure.lang.MultiFn .dispatchFn / .getMethod — spec's multi-spec walks a
multimethod through them.
- (.applyTo f args) applies a fn to a seq of args (spec instrument).
Most of spec.alpha's conform/explain/describe suite passes. Remaining gaps:
explain-data's :pred for a BARE fn predicate (jolt fns don't carry their defining
symbol, so fn-sym can't recover it), #inst form rendering, and instrument — follow-up.
make test green (+3 corpus rows, 0 new divergences), runtime only (no re-mint).
- (proxy [ThreadLocal] [] (initialValue [] body)) now builds a real per-thread
store backed by a Chez thread-parameter, with a lazy initialValue; .get/.set/
.remove work. Other proxies stay nil. test.check's no-seed PRNG (next-rng) uses
one, so gen/sample and gen/generate (and everything built on them) now work.
- clojure.test/*testing-vars* (+ *report-counters*) are bound vars now, so a
defspec run through its :test metadata / default reporter doesn't hit an unbound
var.
make test green (+1 corpus row), shakesmoke byte-identical. One re-mint (proxy).
More general fixes from clojure.test.check's own suite.
- *unchecked-math* on doubles: unchecked-* only wrap integer math; on a flonum
operand they're an ordinary float op (Clojure: (unchecked-multiply 1.5 2.0) =>
3.0). test.check's rand-double is (* double-unit shifted) under *unchecked-math*
and was truncating to a long 0, so every distribution-driven generator (choose,
vector, …) collapsed to its lower bound.
- (take Double/POSITIVE_INFINITY coll) takes the whole coll instead of throwing
on the infinite count coercion (rose-tree unchunk relies on it).
- (java.util.UUID. msb lsb) 2-long constructor (the uuid generator), formatted as
the canonical lowercase 8-4-4-4-12 string; (Long. n) constructor; BigInteger
.shiftLeft / .shiftRight (size-bounded-bigint); number methods now receive args.
- A transient (ITransientSet) responds to .contains / .valAt / .count
(distinct-collection generators).
make test green (+3 corpus rows, 0 new divergences), runtime only (no re-mint).
Two general fixes shaken out by clojure.test.check's own suite (its splittable
PRNG mixes 64-bit longs and binds locals named min/max).
- *unchecked-math* now wraps arithmetic a macro emits. The analyzer rewrote a
bare (+/-/*) to its wrapping unchecked-* under *unchecked-math*, but a macro's
syntax-quote produces clojure.core/* (qualified), which was skipped — so e.g.
test.check's mix-64 multiply grew to a bignum instead of a 64-bit long. The
rewrite now also fires on the clojure.core-qualified form.
- A local binding named like a bare-emitted native op no longer shadows it. ops
where native-ops maps the name to itself (+ - * / < > min max …) emit as the
bare Scheme name; a local `max` emitted the same token, so
(fn [max] (clojure.core/max …)) called the param. munge-name now prefixes such
locals, like reserved words (derived from native-ops so they can't drift).
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (analyzer + backend).
algo.monads' writer monad extends a protocol to clojure.lang.IPersistentList,
but jolt's lists only reported ASeq/ISeq in value-host-tags, so writer-m-add
didn't dispatch ("No method writer-m-add"). jolt models every seq as a list (no
distinct LazySeq — (class (map inc xs)) is PersistentList), so a seq now also
reports PersistentList / IPersistentList / IPersistentStack, in value-host-tags
and host-type-set. extend-protocol clojure.lang.IPersistentList then dispatches
on a list.
algo.monads passes its whole suite (11/11) over tools.macro. Listed in docs +
site. Runtime only, no re-mint. make test green (+1 corpus row, 0 new
divergences), shakesmoke byte-identical.
jolt modelled letfn as a special form directly, so (macroexpand-1 '(letfn …))
returned the form unchanged. Clojure's letfn is a macro that expands to letfn*,
and macroexpansion tooling (tools.macro, tools.analyzer) depends on that — its
special-form handlers key on letfn*, not letfn.
Split it the Clojure way:
- letfn* is now the special form (analyzer), taking flat name/fn-form pairs
[name1 fn1 name2 fn2 …] — the letrec :let lowering is unchanged.
- letfn is a macro (00-syntax) turning each (name [params] body*) spec into a
name + (fn name [params] body*) binding, so it expands to letfn*.
So (macroexpand-1 '(letfn [(f [x] x)] (f 1))) now yields
(letfn* [f (fn f [x] x)] (f 1)), and clojure.tools.macro passes its whole suite
(macrolet / symbol-macrolet / mexpand-all). Listed in docs + site.
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (analyzer + the letfn macro); selfhost holds.
Two general fixes shaken out by clojure/tools.macro.
- The ns macro now accepts a vector reference clause [:require …] / [:use …],
not just the list form (:require …). Clojure dispatches on (first clause) and
accepts both; jolt silently dropped vector clauses, so a ns written with them
loaded with nothing required/used (tools.macro's test ns uses [:use …]).
- clojure.lang.Compiler/specials is now a static whose keys are the special-form
symbols (matching Clojure 1.2/1.3). Macroexpansion tooling reads
(keys Compiler/specials) to know which heads not to expand.
tools.macro itself isn't fully passing yet — its mexpand-all works, but the
macrolet/symbol-macrolet tests need letfn to macroexpand to letfn* (jolt models
letfn as a special form, not a macro over letfn*), so it stays off the list.
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (the ns macro).
data.priority-map's whole suite passes (4/4). It leans on deftype/collection
interop jolt got wrong; four general fixes:
- rseq dispatches to a deftype's clojure.lang.Reversible.rseq method instead of
always demanding a vector/sorted-coll (natives-seq.ss).
- a deftype method declared at two arities from two interfaces now dispatches by
arity: the priority-map has seq[this] (Seqable) and seq[this ascending]
(Sorted), so (.seq pm false) must reach the 2-arg one. find-method-any-protocol
now matches the call's arg count via procedure-arity-mask, and a deftype's own
declared method wins over the generic collection interop in dot-forms.
- (empty x) on a deftype/record with its own empty method uses it rather than
returning {} (jolt.host/jrec-method? gate in clojure.core/empty).
- clojure.lang.Sorted (comparator / entryKey / seqFrom) works on jolt's
sorted-map/set, so subseq/rsubseq run — including the priority-map delegating
.comparator to its backing sorted-map (dot-forms.ss + host-static.ss).
Listed in docs/libraries.md + the site. One re-mint (clojure.core/empty);
everything else runtime. make test green (0 new divergences), shakesmoke
byte-identical.
Completes the JVM long-compatibility gap so clojure.test.check (and the
property-based suites built on it, e.g. data.codec) run on jolt.
A ^long is 64-bit but a Chez fixnum is only 61-bit, so the backend's fast fx
comparison / quot / min / max / inc / dec ops raised on a full-width long (one
from the PRNG or wrapping arithmetic). They now go through the jolt-l* macros
(host/chez/seq.ss): the fx fast path when the operands ARE fixnums, the generic
op otherwise — so e.g. ((fn [^long a ^long b] (< a b)) Long/MAX 1) is false, not
an error. Arithmetic +/-/* keep the raw fx ops (under *unchecked-math* they're
already the wrapping unchecked-*).
Also fixes unsigned-bit-shift-right: it was an arithmetic (sign-propagating)
shift, now a logical shift over the 64-bit two's-complement window, so
(unsigned-bit-shift-right -1 1) is 2^63-1 like the JVM.
Result: test.check 1.1.3 loads and runs (generators, quick-check, shrinking);
data.codec's base64 property suite passes (12/12 defspecs; the 2 deftests check
clojure.lang.IFn$OLLOL, a JVM primitive-fn interface, N/A). Both added to
docs/libraries.md + the site.
re-mint (backend/seed). make test green (+3 corpus rows, 0 new divergences,
numeric gate updated to the jolt-l* ops), shakesmoke byte-identical.
clojure.core's unchecked-* (and +/-/*/inc/dec under *unchecked-math*) are long
ops that WRAP on overflow; jolt's checked arithmetic is arbitrary-precision and
its unchecked-* were plain non-wrapping (+ x y), diverging from the JVM. Now they
truncate to the low 64 bits as a signed long, matching Clojure:
(unchecked-add 9223372036854775807 1) => -9223372036854775808
(unchecked-multiply 9223372036854775807 …) => 1
- host/chez/seq.ss: jolt-wrap64 + binary jolt-unc{add,sub,mul,inc,dec,neg}2 and
the variadic clojure.core/unchecked-* fns (def-var!'d in natives-seq.ss, where
def-var! is bound). The overlay's plain unchecked-* defns are removed.
- backend lng-ops: unchecked-+/-/* emit the wrapping jolt-unc* helpers (the
raising fx ops can't wrap on Chez's 61-bit fixnums); unchecked-inc/dec too.
- *unchecked-math* is honored: the analyzer reads it (jolt.host/unchecked-math?)
and rewrites +/-/*/inc/dec to their unchecked-* for the rest of a file that
(set!)s it, like the JVM.
- jolt->fx: a ^long value that overflows the 61-bit fixnum range passes through
as an exact integer instead of erroring (a full-width long from wrapping math).
Also adds Long/bitCount / numberOfLeadingZeros / reverse and Math/getExponent /
scalb (test.check's splittable PRNG uses them).
This lets clojure.test.check load and run quick-check on jolt. re-mint (analyzer/
backend/overlay are seed sources). make test green (+6 corpus rows, 0 new
divergences, numeric gate updated), shakesmoke byte-identical.
clojure.data.csv runs its whole suite on jolt (4/4 reading/writing/eof/line-
endings). Three general gaps fixed, all runtime, no re-mint, JVM-certified:
- The prefix-list form of :require/:use — (:require (clojure [string :as str]))
means clojure.string :as str — now expands (loader.ss). It silently failed
before, trying to load a "clojure" namespace.
- extend-protocol to java.io.Reader / Writer / StringReader / PushbackReader now
dispatches: those reader/writer host tags carry the right class names in
value-host-tags AND are in host-type-set, so extend-protocol registers under
the canonical tag instead of a local ns tag (records.ss). data.csv's
Read-CSV-From protocol extends to String / Reader / PushbackReader.
- (str StringWriter) returns its accumulated content (register-str-render for the
"writer" jhost), not the opaque host object — data.csv writes CSV to one and
reads it back.
Listed in docs/libraries.md + the site.
make test green (+2 corpus rows, 0 new divergences), shakesmoke byte-identical.
clojure.zip was missing xml-zip — a zipper over xml {:tag :content} elements,
which clojure.data.zip and any xml-zipper code needs. Added (runtime, loaded on
require). clojure.data.zip's whole xml suite (9/9) then passes, once XML parsing
is provided: clojure.xml/parse now ships in jolt-lang/xml over its
javax.xml.stream pull parser (committed there).
Listed in docs/libraries.md + the site.
clojure.core.contracts (over core.unify) now runs its whole suite on jolt —
14/14 across contracts/constraints/with-constraints/provide tests. Two general
gaps fixed:
- Symbol and Keyword now report IFn (and Fn/Runnable/Callable) in the modeled
class hierarchy, so a (class x)-dispatched multimethod with an IFn method
matches a symbol or keyword, like the JVM (both implement IFn — they're
callable). core.contracts' funcify* dispatches on (class constraint) and a
bare predicate symbol must hit the IFn arm. Runtime, no re-mint.
- A live Var value spliced into a form by a macro (defcurry-from resolves a var
and emits (~v l r)) now compiles: analyze treats a var-cell form as a
:the-var reference by ns+name, the same node as (var ns/name), mirroring the
existing spliced-namespace (~*ns*) case. analyzer.clj + host-contract.ss,
re-mint (prelude stays byte-identical; only the analyzer image changes).
Listed in docs/libraries.md + the site.
make test green (+2 corpus rows, 0 new divergences), shakesmoke byte-identical.
Running clojure.tools.reader's own suite on jolt surfaced a batch of general
gaps (all runtime, JVM-certified, no re-mint — reader.ss is loaded at runtime
and jolt-core has no octal literals, so selfhost holds):
Reader:
- (load "rel") resolves a non-/ path against the current namespace's directory,
like Clojure — (load "common_tests") from clojure.tools.reader-test loads
clojure/tools/common_tests.clj. Was resolved against the roots directly.
- Octal integer literals: 042 reads as 34, not decimal 42; octal string escapes
(\377 is one char, not \0 + "00"). \oNNN char octal already worked.
- (symbol nil name) now equals (symbol name) and the reader literal — a nil
namespace is the #f no-ns sentinel, not jolt-nil (jolt= compares ns by equal?).
clojure.test:
- thrown-with-msg? honors the class hierarchy (instance?) before falling back to
a simple-name match, so (thrown-with-msg? RuntimeException ...) matches an
ExceptionInfo, like thrown? already did.
Host interop (java layer):
- java.util.regex: Pattern.matcher / Matcher.matches / .group / .groupCount /
.find, and Pattern/compile.
- clojure.lang: RT/map, PersistentList/create, PersistentHashSet/createWithCheck.
- java.lang.Character: digit / isDigit / isWhitespace / valueOf.
- java.util.LinkedList (Deque surface over the ArrayList backing); ArrayList /
LinkedList are now seqable.
- BigInteger 2-arg ctor (string, radix) + .negate / .bitLength / .signum / .abs;
BigInt/fromBigInteger and Numbers/reduceBigInt (identity on jolt's exact ints).
Suite: reader_test 22/30, reader-edn_test 13/16. The remaining failures are
fundamental numeric-model differences (no BigDecimal type; BigInt and Long are
one exact-integer type) or need JVM reflection (record/ctor tagged literals via
getConstructors) — out of scope.
make test green (+8 corpus rows, 0 new divergences), shakesmoke byte-identical.
Running clojure.core.typed's runtime contract tests (typed/runtime.jvm,
test_contract — 5/5 pass) surfaced two general jolt gaps, both runtime, both
JVM-certified:
- instance? Object / java.lang.Object returned false for everything. Object is
the root of the type hierarchy: every non-nil value is an instance of Object,
nil is not. core.typed's (instance-c Object) contract depends on this; many
libraries do.
- @Compiler/LINE and @Compiler/COLUMN (clojure.lang.Compiler statics — Vars on
the JVM holding the line/column of the form being compiled) were unresolved.
Macros read @Compiler/LINE as a fallback when &form carries no position. Now
backed by derefable cells updated per top-level form, like *current-source*.
The core.typed type checker itself (tools.analyzer.jvm + ASM bytecode +
clojure.lang.Compiler internals) and the cljs runtime are not portable, so the
checker/check-ns surface is out of scope; this is the runtime contract layer.
make test green (+4 corpus rows, 0 new divergences), shakesmoke byte-identical.
EPL-1.0 is superseded by EPL-2.0 (clearer jurisdiction/patent terms). Updates
the LICENSE file and README. Clojure-derived files under stdlib/ and the
vendored sci keep their original EPL-1.0 headers per the weak-copyleft terms.
Closes#206
Adds clojure.core.async's higher-level dataflow API as a Clojure overlay
(stdlib/clojure/core/async.clj) over jolt's native channel primitives, plus
clojure.core.async.lab. The native layer (host/chez/java/async.ss) gains
offer!/poll!, put specs and :priority/:default in alts!, a transducer
ex-handler arg to chan, unblocking-buffer?, promise-buffer, and on-caller?
handling for put!/take!. The overlay covers alts!/pipe/pipeline/split/
reduce/transduce/into/take/mult/mix/pub-sub/map/merge/onto-chan/to-chan and
the deprecated map</map>/filter>/... family (rewritten as go-loops since the
JVM versions reify the impl handler protocol jolt doesn't expose).
Loading: the native primitives pre-seed clojure.core.async, so the loader now
drops it from the loaded set and a require pulls the overlay from the source
roots like clojure.test (AOT-bundled into built binaries).
Running clojure/core.async's own suite shook out two general bugs:
- :refer with a list form, (:require [ns :refer (a b c)]), dropped the names
(only the vector form was handled) — chez-register-spec! now accepts both.
- (range 0) / (range 5 5) returned nil instead of the empty seq () — empty
ranges now match Clojure, so (= () (range 0)) holds.
Suite: async_test 15/20, pipeline_test 7/7, timers_test 2/2, lab_test 2/2.
The five non-passing async_test cases all assert JVM go-machine limitations
jolt's thread-based model is a superset of (the 1024 pending-op cap, parking
ops that must throw outside a go block, expanding-transducer buffer
backpressure) or dispatch-thread identity, not data semantics.
make test green (0 new divergences, +4 range corpus rows), shakesmoke
byte-identical.
A macro body can now read &form (the call form) and &env (a map of the in-scope
local symbols), like Clojure. This is what core.logic's matche/defne use to tell
a pattern symbol that names an enclosing local from a fresh pattern var — so
locals-membero and the recursive checko in `matches` now compute correctly. The
suite reaches 535/2/0 (the last two are constraint reification ORDER, where the
constraint set is right but it is spliced from a set whose iteration order differs
from the JVM — a host set-ordering divergence, not a bug).
&form/&env are clojure.core dynamic vars bound around each expander call rather
than prepended params, so the macro calling convention is unchanged and the mint
stays consistent (the seed prelude is byte-identical; only the analyzer carries
the env into form-expand-1). macroexpand-1 passes an empty env.
corpus.edn: the ~@ unquote row is now a boolean compare (a bare clojure.core/
unquote-splicing symbol evaluates to an unbound var, not the symbol).
Two general fixes that clear core.logic's finite-domain -difference, safefd, and
the defne quoted-list patterns (form->ast), taking the suite to 532/5/0.
- instance? on a deftype matched a simple type name against the qualified tag by
raw string suffix, so "a.b.MultiIntervalFD" tested true for IntervalFD. The
suffix must land on a "." boundary. core.logic's fd dispatches on
(interval? x) = (instance? IntervalFD x), and a MultiIntervalFD wrongly counted
as an interval, so -difference/safefd computed the wrong set.
- the reader reads ~ / ~@ as clojure.core/unquote(-splicing), like the JVM reader,
instead of a bare unquote. Code that inspects quoted pattern/template data —
core.logic's defne checks (= f 'clojure.core/unquote) — now sees the symbol it
expects, so '(fn ~args . ~body) patterns compile. hc-head-is? accepts the
qualified head in syntax-quote lowering; the value-preserving change leaves the
minted seed byte-identical.
corpus.edn: 2 JVM-certified unquote rows. unit.edn: two reader rows updated to the
qualified unquote. make test + shakesmoke green, 0 new divergences, self-host holds.
value-host-tags returned only ("Object") for a record, so a protocol extended to
clojure.lang.IRecord / IPersistentMap / Associative / Seqable / … (and not to the
record's own type) dispatched to the Object default. core.logic extends IWalkTerm
to IRecord, and walking a record value hit Object's (walk-term [v f] (f v)) — which
re-enters walk* and loops forever (the test-53-lossy-records hang).
A defrecord now carries the map/record interface tags it genuinely satisfies. The
record's own type is still tried first (jrec-tag, before these tags), so a direct
extend to the record type wins, and record equality / map? / record? are unchanged.
A bare deftype stays opaque (its tag + Object; declared interfaces dispatch via its
inline methods). Runtime only, no re-mint.
A name a namespace excludes from clojure.core (:refer-clojure :exclude) is not
clojure.core/name even before the ns defines its own — syntax-quote must qualify
it to the current ns, like Clojure. refer-clojure was a no-op, so a syntax-quoted
excluded name (core.logic.fd's `==`, referenced by a constraint's -rator before fd
defines ==) resolved to clojure.core/==.
jolt-refer-clojure now records the :exclude set per ns; hc-sq-symbol consults it
before falling back to clojure.core. Fixes core.logic's fd constraint -rator names.
Runtime only, no re-mint.
Follow-on to the core.logic relational-engine work. These clear every crash in
core.logic's constraint-logic-programming and unifier layers (33 errors -> 0) and
most of the value mismatches; the suite goes 504 -> 523 passing assertions. All
are general gaps, not core.logic-specific.
- symbols intern their ns/name strings (JVM Symbol.intern .intern()s them): two
separately-read `?a` symbols now share one name-string object. core.logic's
non-unique lvars compare names by identity (via (str sym)), so without this a
term's lvar and a constraint's lvar built from different `?a` reads never matched
and constraints silently never fired.
- (str x) of a single arg returns its rendering directly instead of copying through
string-append, and a symbol stringifies to its (interned) name — JVM (str x) is
x.toString(). Needed for the identity comparison above.
- a clojure.core-qualified special form dispatches correctly: syntax-quote
namespace-qualifies a macro like letfn to clojure.core/letfn (matching Clojure,
where it's a macro), and the analyzer now maps that back to the special form
instead of treating it as an invoke of a nil var. core.logic's fnc/defnc emit
(clojure.core/letfn ...). Re-mint.
- (disj nil ...) is nil (JVM), instead of crashing in the set path — core.logic's
constraint store does (disj (get km v) id) where the get can be nil.
corpus.edn: 4 JVM-certified rows. make test + shakesmoke green, 0 new divergences,
self-host fixpoint holds.
Running clojure/core.logic's own suite surfaced a batch of general jolt gaps.
None are core.logic-specific; each is a language/host behavior that was wrong or
missing. With these, the core relational engine (unify, run/fresh/conde,
conso/membero/appendo, reification to _0/_1, lcons) runs; the remaining failures
are in core.logic's constraint-logic-programming and finite-domain layers
(tracked separately).
- analyzer: accept the list-member dot form (. target (method args)), sugar for
(. target method args). Re-mint.
- identical? is reference identity (eq?), not value equality. It was aliased to =,
which infinite-loops when a deftype's .equals short-circuits on (identical? this o)
(core.logic's Substitutions) and is wrong for distinct equal collections.
- jrecs use a deftype's declared hashCode/equals/equiv for map/set keying instead
of structural field comparison, so metadata-wrapped keys still match (core.logic
keys substitutions on lvar id, ignoring metadata).
- meta/with-meta dispatch to a deftype's clojure.lang.IObj meta/withMeta methods
when present, so metadata threaded through the type's own assoc/withMeta survives
(previously kept in an identity side-table the reconstructed instances didn't share).
- coll?/seqable? on a deftype require IPersistentCollection (cons) or ISeq (first);
ILookup(valAt)/Indexed(nth)/Counted(count)/Seqable(seq) alone no longer qualify,
matching the JVM.
- syntax-quote resolves a bare symbol to the compile ns's own def before
clojure.core, so a name the ns excluded and redefined (core.logic's == after
:refer-clojure :exclude) qualifies correctly in macro output.
- reader: record literals #ns.Type{...} / #ns.Type[...] expand to the map->/->
factory call.
- structmap API: defstruct/create-struct/struct-map/struct/accessor (map-backed,
insertion-ordered). Re-mint.
- .hashCode on strings/symbols (Java String.hashCode, Symbol Util.hashCombine);
Class.isInstance; java.util.Collection.contains over vector/list/set;
clojure.lang.RT/nextID and clojure.lang.Util hash/hasheq/equiv/identical statics.
corpus.edn: 8 JVM-certified rows. unit.edn: a Counted+Seqable deftype is coll?=false
(was a stale expectation encoding the old behavior).
jolt maps were HAMTs with hash iteration order; Clojure keeps small maps as
PersistentArrayMap (insertion order), converting to PersistentHashMap past a
threshold. Map literals, array-map, assoc, into/transient, merge, zipmap,
select-keys, update-keys/vals, frequencies and group-by now iterate in insertion
order for <=8 entries, matching the JVM. hash-map and >8-entry maps stay hash
order; sets stay hash order.
The pmap record gains an order field (the insertion-order key list, or #f once
hashed); the HAMT still backs the values so equality/hash/lookup are unchanged.
pmap-fold visits an array-mode map last-to-first so the runtime's cons-accumulate
idiom reconstructs insertion order without touching its many call sites, and
hash-mode output stays byte-identical; pmap-fold-fwd visits in order for the few
sites that build a value directly. Transient maps track insertion order and
promote to hash past max(8, source-count), matching TransientArrayMap.
The hash-map native-op retargets to a hash-order builder so (hash-map ...) stays
hash-ordered while {...} literals are ordered; syntax-quote builds maps via the
hash builder (Clojure expands `{...} to apply hash-map). The core overlay map
builders seed from {} instead of (hash-map) to keep order.
Threshold is 8 for any key (the keyword exception in newer Clojure isn't in
1.12.5). honeysql now passes 832/0/0; 19 JVM-certified corpus rows added.
defn- now adds :private to the var metadata (like Clojure), and ns-publics
filters those out while ns-interns/ns-map keep them — they were all the same
unfiltered scan before. A lib that introspects ns-publics (honeysql asserts
every public helper has a docstring, and that the clause set matches the public
helpers) saw the private defn- helpers and failed; now honeysql 636/8 -> 638/6
(the rest are map key-order).
jolt's case ops are codepoint-based and locale-independent, so the default
locale is a no-op token: getDefault/setDefault/forLanguageTag + ROOT/US/ENGLISH.
honeysql sets and restores the locale around formatting to assert output is
locale-stable (its Turkish-İ regression guard) — that test errored on the
missing Locale/setDefault static, now passes (honeysql 635/8/1 -> 636/8/0).
clojure.edn/read built the built-in #inst/#uuid eagerly (via read-string), so a
:readers override couldn't win and #inst applied to a non-string form (aero's
#inst ^:ref […]) threw. Read the raw form instead and let edn->value route every
tag through :readers then :default then the built-in — matching clojure.edn,
where a reader from opts wins. edn->value now also converts the (recursively
converted) metadata, since the raw path skips the read-string data seam. aero
suite: 59/0/0 (full pass). clojure.edn baked, re-minted.
meta-copy keyed metadata on the SAME cseq/lazyseq object (the else branch), so
(with-meta xs m) mutated the original list in place — Clojure's PersistentList
is immutable and withMeta returns a new list. (with-meta xs {:k xs}) thus built
a self-referential cycle (the list's metadata pointed at the list), which looped
*print-meta* printing forever — the root of aero's meta-preservation hang. Now
copies the cseq/lazyseq node like the other collections. aero suite completes:
58/0/1 (was hanging).
clojure.edn/read with :readers/:default recursed into vectors/maps/seqs but
not a constructed set, so a tagged literal in #{…} (aero's #ref in a set) kept
its raw form. edn->value now recurses into a set. clojure.edn is baked into the
seed, re-minted. Fixes aero #ref-in-set + falsey-user-return.
Clojure's reader reads a ^{…} map with the same read() as any value, so a set/
tagged literal in metadata is a value, not a form. jolt's data seam converted a
set-form to a set in the VALUE but left it as the tagged form inside the
METADATA, and dropped metadata on an empty list entirely (a wrong 'interned, =
Clojure' special case in rdr-attach-meta — Clojure's MetaReader withMetas () via
IObj). rdr-form->data now always converts + carries the (recursively converted)
metadata, whether or not the value structure changed; rdr-attach-meta no longer
skips (). Fixes aero's meta-preservation (set/map/vector/empty-list ds round-
trip). All runtime .ss (data seam), no re-mint.
Two gaps reitit-core surfaced (now 322/0/1 -> 327/0/0):
- A deftype/defrecord inline method with two _ params, (m [_ _] field), read
the field as nil: mk-clause bound fields off (get _ :field) where _ was the
first param, but the second _ shadowed it. Each _ param is now renamed to a
fresh symbol so the instance is unambiguous.
- A var did not dispatch to a protocol's clojure.lang.Var extension (reitit
extends Expand to Var for a #'handler route): value-host-tags gained a var arm
(Var/clojure.lang.Var/IDeref/IFn) and host-type-set gained Var/IDeref so the
extension keys under Var.
deftype/defrecord is a seed source, re-minted.
jolt had Long/Integer/Double class tokens but not Byte/Short/Float, and no
Byte/Short MIN_VALUE/MAX_VALUE/valueOf/parse* statics. clojure.test.check (a
malli dependency) references Byte/MIN_VALUE and Byte/MAX_VALUE. The values are
plain integers on jolt; the statics expose the JVM ranges (127/-128, 32767/
-32768).
A defrecord is Associative/ILookup/IPersistentMap/Seqable/Counted on the JVM,
so (.assoc r k v) / (.valAt r k) / (.without r k) / (.containsKey r k) /
(.cons r x) / (.count r) / (.seq r) / (.equiv r o) / (.entryAt r k) now work
via Java interop, delegating to the map fns when not overridden by a declared
method. reitit's impl calls (.assoc match k v) directly. A bare deftype uses
its own declared methods (record-only branch). reitit-core 58/1/18 -> 321/1/1
with the router lib's Trie shim.
#{...} reads as the tagged set-form for the analyzer, but a macro saw that
map instead of a set (set? false / map? true, unlike a vector). hc-expand-1
now converts a set-form argument to a real set before calling the expander, so
(set? arg)/conj/seq work — hiccup's compiler introspects a literal set this way
(str (html #{"<>"}) was empty, now #{"<>"}). Elements stay as
read; a deeply-nested set literal inside another form is left for the analyzer.
hiccup 382->383. Jolt-side unit guards (macro def+use in one form isn't
JVM-portable).
(defn name docstring? {:k v} arglists...) and the multi-arity name+attr-map
now merge the attr-map into the var metadata like Clojure — jolt was parsing
the map out of the body and discarding it. The metadata (the name's own ^{},
the attr-map, and the docstring as :doc) is attached to the def name symbol,
which analyze-def reads and evaluates. defn is in the earliest tier, so the
macro uses only conj/assoc/meta/with-meta (not merge/last). The rare trailing
attr-map (after the last arity) is not yet handled. Fixes hiccup's defelem
meta + honeysql docstring tests.
('sym coll) / ('sym coll default) now do (get coll 'sym ...), like keywords —
a symbol is IFn on the JVM. jolt threw "cannot be cast to clojure.lang.IFn".
Pre-existing gap (not a regression), surfaced by honeysql's :checking mode,
which does ('where dsl) to look up a clause. honeysql 623/13/8 -> 635/8/1.
Corpus rows added.
The deftype-is-not-a-map change (#245) gated map? on jrec-record?, so only a
defrecord was map?. But a deftype that implements clojure.lang.IPersistentMap is
map? on the JVM — clojure.core.cache's caches are exactly that, and its TTL
factory asserts (map? base) on an LRUCache passed as the base (its suite went
1314 -> 2 errors). map? now also covers a deftype whose without/dissoc method is
registered — the IPersistentMap-distinctive op a vector or set lacks. An opaque
deftype (RawString) stays non-map?; a defrecord stays both. Guards added to
unit.edn (jolt-side: a full IPersistentMap impl will not compile on the JVM
corpus oracle).
The rest = more() change made (rest coll) return a jolt-lazyseq, so the very
common (conj (rest xs) y) hit jolt-conj1's base case, which doesn't recognize a
lazyseq, and threw "conj: unsupported collection" (caught by core.match's
seq-pattern compiler). conj on a lazy-seq now prepends like conj on any seq.
The corpus had no row exercising a collection op on a rest-derived seq, so the
class slipped past the gate; add a seqs/lazy-seq-interop suite (conj/into/first/
count/nth/reduce/map/filter/apply/cons/=/empty?/seq over (rest …) and lazy-seq),
all JVM-certified.
- with-deeper-print only parameterizes the print depth when *print-level* is
set, so printing pays no parameterize on the common nil-default path.
- re-find (the matcher) and re-seq advance past a zero-width match relative to
the match's own start, not the search origin — a zero-width match found past
the origin (lookahead/boundary) no longer repeats. (re-seq #"a*" "aba") now
matches the JVM.
A library that throws a typed host exception (an http client raising
java.net.ConnectException) needs (class e), instance?, .getMessage and
ex-message to all reflect the named class. jolt-host-throwable already builds
that value; expose it as a jolt.host seam so libraries stop hand-rolling a
:jolt/ex-info table that carries only the class (its .getMessage/ex-message
return nil).
Six correctness fixes, each a general gap (not hiccup-specific):
- deftype is not a map. jolt treated every deftype instance as a map
(map?/record?/seqable over its fields); in Clojure only a defrecord is
map-like, a bare deftype is an opaque object. defrecord now marks its type;
map?/record?/coll?/seq/empty? gate on it, while a deftype implementing a
collection interface still dispatches through its methods.
- cross-ns extend-protocol on an imported deftype. register-method built the
type tag from the *calling* ns + bare name, so (extend-protocol P Raw …) in
one ns missed a Raw value defined in another. A simple-name index resolves
the bare name to the type's real tag (local ns still wins).
- str vs print. str of a collection is its readable form (nested strings
quoted: (str ["x"]) => ["x"]); print leaves them raw. jolt defined print
as str, conflating the two. Split via a __print1 seam.
- clojure.test thrown? now honors the exception hierarchy (instance?), so
(thrown? IllegalArgumentException …) matches an ArityException subclass.
- java.net.URI is value-equal (= and hash by string form).
- clojure.walk/macroexpand-all was missing; an unresolved qualified var made
the analyzer report "Unknown class walk".
deftype/defrecord + print are seed sources, re-minted. hiccup 365->381 of its
own suite; the rest are charset-encoding / var-meta niches.
jolt's seq layer realized one element ahead of Clojure, so a side-effecting
lazy seq ran its producer too eagerly. Four changes bring it in line:
- rest is Clojure's more(): it returns the tail without realizing it. An
unforced tail (vector / string / lazy-seq cell) comes back as a deferred
seq, so (rest (iterate f x)) does not call f. next still realizes one.
- iterate applies f lazily, inside the tail thunk, so (first (iterate f x))
is x with no call to f (clojure.lang.Iterate parity).
- take realizes exactly n: the last element terminates without touching the
rest, instead of forcing one more element of the source.
- an empty realized lazy seq is still a sequence value, printing "()" not
"nil" (a JVM LazySeq is never nil).
Also: the map transducer's step fn now takes multiple inputs
([result input & inputs]) so a multi-collection transduce applies f across
all of them. Fixes medley's join/window/sequence-padded laziness and
multi-input transducer tests (now 293/293). The rest change also fixed a
latent overrun in distinct/dedupe over a map's empty tail.
iterate is a seed source, re-minted.
Both printers (jolt-pr-str, jolt-pr-readable) now thread a print depth and
read the two limit vars. *print-length* truncates each collection to N
elements + "...", walking seqs lazily so an infinite seq prints under the
limit without realizing it. *print-level* renders a collection at depth >=
the level as "#". The reader consults *default-data-reader-fn* for an
unregistered #tag before falling back (tagged form on the data seam, throw
on the edn seam). All three interned with nil defaults.
A wrong-arity or non-seqable error that Chez raises carried no jolt exception
class, so (class e) was :object and (thrown? ArityException …) / (thrown?
IllegalArgumentException …) never matched. Classify these by message:
incorrect-number-of-arguments -> clojure.lang.ArityException, not-seqable ->
java.lang.IllegalArgumentException, with ArityException modeled as a subclass of
IllegalArgumentException. (class e) and instance? now match the JVM.
All java-layer (records-interop.ss classifies, host-class.ss reports the class +
the ArityException token). medley 281 -> 283 passing.
jolt-o9dc
Spec coverage dashboard had 6 missing-portable and 24 dynamic-var entries. The
portable ones are now implemented (missing-portable -> 0, dynamic-var -> 14):
- Stateful matcher: re-matcher now returns a real mutable Matcher; re-find over
it steps through matches and re-groups returns the last match's groups (was an
inert tagged map). Closes re-groups.
- letfn is interned as a clojure.core var so (resolve 'letfn) matches the JVM. It
stays a special form (the value is never invoked, not marked a macro).
- *1 *2 *3 *e interned (nil outside a REPL).
- Portable dynamic vars whose default already matches jolt's behaviour:
*read-eval* *print-dup* *print-namespace-maps* *flush-on-newline*
*compile-files* *math-context* *command-line-args* *file*.
The remaining 14 dynamic-var entries are host-internal (compile-path,
compiler-options, fn-loader, reader-resolver, repl, source-path, ...) or deferred
pending printer/reader support (*print-length* *print-level*
*default-data-reader-fn*). Corpus rows added for each closed gap; coverage.md
regenerated.
irregex rejected two patterns the JVM accepts, which blocked library loads:
- [\w-_] errored with bad char-set because a - after a shorthand class was
read as a range start. Java reads it as a literal hyphen. Preprocess the
pattern to escape such a dash.
- (X+)* errored with duplicate repetition because sre-repeater? recurses
through submatch, treating a quantified group like a dangling a**. Override
it to a bare leading * / + check, matching the JVM (which only rejects the
dangling case).
Both in regex.ss (runtime). Unblocks cuerdas (was load-fail, now 292 passing)
and aws-api config-test. Also documents the host/chez/java source-layering rule
in host-interop.md.
jolt-l8so
(type x) was jolt's internal taxonomy keyword (:string/:set/:jolt/inst), which
breaks any library dispatching a multimethod on [(type a) (type b)] against
java/clojure.lang classes (e.g. clojure.tools.logging.test's matchers). Make the
PUBLIC clojure.core/type Clojure's (or (:type meta) (class x)).
The taxonomy keyword stays the core model: natives-meta.ss keeps jolt-type and
exposes it as __type-tag, which print-method/print-dup dispatch on (so #uuid/#regex/
records still print). The JVM mapping lives in the java host layer — host-class.ss
defines the public type next to (class …), and a jinst now reports java.util.Date
(was :jolt/inst). So the core emits the taxonomy and the java layer remaps it in one
place. unit.edn's type suite updated to the class names. make test green.
Co-authored-by: Yogthos <yogthos@gmail.com>
Class/forName claimed every java.*/clojure.* name found (and any "x.y.Class"
matched the registered Class via a short-name fallback), so a library's
(class-found? "optional.Dep") feature-probe always said yes — tools.logging then
tried to build the java.util.logging / log4j backends jolt lacks and crashed.
Resolve forName by exact registry lookup + an honest prefix that excludes the
unbacked optional packages (java.util.logging, javax.management), so the probe
sees them absent and skips the backend.
class of a persistent collection / namespace now reports its JVM class name
(clojure.lang.PersistentHashSet, …Namespace, …) instead of jolt's internal :set/
:object tag, and isa? consults JVM class assignability — Object as every class's
root plus a modeled clojure.lang/java.util hierarchy — so (isa? (class x) C) and a
class-keyed multimethod dispatch like the JVM (e.g. (isa? Keyword Object) was
false). Adds the bare class tokens (Fn/Namespace/Set/…) these dispatch on.
(type x) is unchanged — it keeps jolt's documented internal-keyword form. Six
JVM-certified corpus rows. make test green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
A library that throws an ex-info envelope carrying a JVM :class (jolt.host/
tagged-table with a "class" entry — e.g. http-client's UnknownHostException on a
DNS failure) was caught only by a broad (catch Throwable …): (class e) read the
class but (instance? C e) — which catch dispatch lowers to — always returned
false, so clj-http-lite's (catch UnknownHostException e …) for :ignore-unknown-host?
never matched and the condition propagated.
Add an instance? arm matching such an envelope against the carried class or any
ancestor (full name or last segment), and register the common exception hierarchy
(Throwable/Exception/RuntimeException/IOException + the java.net socket/host
exceptions) so (catch IOException e) / (instance? Throwable e) also match. A
non-throwable class (RuntimeException over an IOException, String) stays false.
Fixes http-client's :ignore-unknown-host? test (116/0/0, was 1 error). make test
green, 0 new divergences. Runtime .ss, no re-mint.
Co-authored-by: Yogthos <yogthos@gmail.com>
A value class libraries return from getters (java.time, and the java.net.http
client shim that aws-api's java backend builds on). Statics of/ofNullable/empty,
methods isPresent/isEmpty/get/orElse/orElseGet/ifPresent/toString, value-equal so
(= (Optional/of x) (Optional/of x)). Five JVM-certified corpus rows. Runtime .ss,
no re-mint.
Co-authored-by: Yogthos <yogthos@gmail.com>
parse-ms ignored sub-second fractions: a formatter parse of
"2020-07-06T10:59:13.417Z" dropped the .417 (and the ISO_OFFSET_DATE_TIME pattern
"…ssXXX" then mis-aligned, reading ".417Z" as the offset). java.time's ISO
formatters accept an optional fractional second, so jolt should too.
After parsing the seconds field, consume an optional .fff from the input (to
millis) when the pattern carries no fraction field — which is how the ISO_*
constants are modeled here (ss, no S). Also handle the S pattern letter for
explicit .SSS patterns. Carry the fraction into the result.
Fixes aws-api shape iso8601 parse (1 FAIL -> 0; cognitect.aws.util/parse-date now
returns the right Date). Two JVM-certified corpus rows. make test green, 0 new
divergences. Runtime .ss — no re-mint.
Co-authored-by: Yogthos <yogthos@gmail.com>
tick's fields-test walks every ChronoField a temporal supports and reads it,
which crashed on fields jolt didn't implement. Fill the gaps:
- LocalTime: CLOCK_HOUR_OF_DAY (1..24), HOUR_OF_AMPM, CLOCK_HOUR_OF_AMPM,
MICRO_OF_DAY — both isSupported and getLong.
- LocalDate: the aligned-* group (ALIGNED_DAY_OF_WEEK_IN_MONTH/_YEAR,
ALIGNED_WEEK_OF_MONTH/_YEAR).
- LocalDateTime field routing now asks which part supports the field instead of a
hardcoded date list, so a date field never misroutes to the time part (the actual
cause of "LocalTime has no field ALIGNED_DAY_OF_WEEK_IN_MONTH" — a ZonedDateTime's
date field fell through to its time).
- Year / YearMonth gain isSupported / get / getLong.
tick api_test 344/0/1 -> 599/0/0. Seven JVM-certified corpus rows. make test green,
0 new divergences. Runtime .ss — no re-mint.
Co-authored-by: Yogthos <yogthos@gmail.com>
The per-site inline cache (#237) resolves a statically-proven monomorphic devirt
once instead of per call, so mono-dispatch is no longer worse than megamorphic.
The remaining dispatch lever is the megamorphic case (a runtime receiver-type-keyed
cache).
Co-authored-by: Yogthos <yogthos@gmail.com>
A devirtualized protocol call resolved its impl with devirt-resolve on EVERY call
— but the tag/proto/method are compile-time constants, so the resolved fn is a
runtime constant (closed world). That per-call find-protocol-method (three
hashtable lookups) was the cost: on mono-dispatch, dispatch was ~75% of the time
(ablation: same arithmetic direct-call 166ms vs dispatch 673ms).
Resolve once. When emitting a direct-link def, each devirt site gets a fresh cache
cell, bound to #f in a let wrapping the def (so it persists across calls and is
shared by every invocation); the site resolves into it on first use ((or cell
(let ((_f (devirt-resolve ..))) (set! cell _f) _f))) and reuses it after — the
inline cache the JVM gets for free. First call still passes the real receiver, so
the Object/host-tag fallback (devirt-resolve) is unchanged.
mono-dispatch 673ms -> 214ms (~3.15x), 47.5x -> ~15x JVM, near the 166ms
direct-call floor. run-devirt.ss gains the cached-path checks (cell present, 1st
call caches + 2nd reuses, both == dispatch). make test / shakesmoke green, selfhost
holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
The type-proving / native-record / bare-field-read / inference work collapsed
the dispatch + allocation gaps by an order of magnitude (binary-trees ~140x->~10x,
mono-dispatch ~330x->~48x, dispatch ~130x->~12x) and brought compute to parity
(fib now beats JVM, collections ~4x, mandelbrot ~7.5x). mono-dispatch is now the
standout gap — a runtime-monomorphic call site the JVM inline-caches to near-free.
Co-authored-by: Yogthos <yogthos@gmail.com>
A record-or-nil (a protocol method whose impls return a record in one branch and
nil in another, or an `if` over a ctor and nil) now types as a NILABLE record
instead of widening to :any. A nilable record still bare-indexes its field reads
(jrec-field-at falls back to jolt-get on nil), but some?/nil? do NOT fold on it, so
a runtime guard is preserved — and inside (if (some? x) ..) / (if x ..) the then-
branch narrows x to the non-nil record, so its reads bare-index AND unbox there.
This is what lets the bounced ray type without a hint: scatter returns
ScatterResult-or-nil (Metal absorbs some rays), and the consumer reads
(:ray scattered) only under (if (some? scattered) ..). The narrowing proves
scattered non-nil there.
lattice: :nil type; :nil ∨ struct -> nilable struct, ∨ anything else -> :any;
nilability is contagious through a struct join, which also now preserves :type when
both sides agree (needed so a record ∨ its nilable self stays that record).
truthy-type?/field-type/pred-on treat a nilable struct as maybe-nil. types: nil
literal -> :nil; an `if` whose test is (some? x)/(nil? x)/x narrows the nilable
local x in the proven branch.
Ray tracer with NO hints: 38.4s -> 23.9s (~1.6x) — hit-sphere now types fully
(0 jolt-get, 57 jrec-field-at, 38 fl-ops), identical to the hand-hinted build.
run-narrow.ss gate, incl. the load-bearing check that the nil case still takes the
else branch (the guard is not folded away). make test / shakesmoke green, selfhost
holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method whose impls all return the same record type has a monomorphic
return. collect-pm-rets! scans the unit's (register-(inline-)method ..) forms,
infers each impl fn's return type, and joins them per method; call-ret-type then
types a (method recv ..) call as that record, so a field read off the result
bare-indexes — e.g. (:ray (scatter m ..)) reads off a Ray. A disagreeing impl
joins to :any and keeps the generic path.
run-protoret.ss: a method with all-record impls bare-indexes + unboxes the field
read; a mixed-return method (one impl returns a number) stays generic. make test /
shakesmoke green, selfhost holds, 0 new divergences.
Foundation for auto-typing record values that flow through protocol dispatch. Does
not yet move the ray tracer: its scatter returns ScatterResult-or-nil (Metal
absorbs some rays), and the nil widens the join to :any — typing a nullable return
soundly needs flow-sensitive narrowing (a guarded (some? x) proves non-nil), filed
separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
The whole-program fixpoint collects a self-recursive call's arg types into the
fn's own params. When a recursive call threads a param straight through unchanged
(same arg, same position — e.g. ray-cast passing `hittables` to itself), that arg's
type is the param's own current type: :any until external callers determine it. And
:any is absorbing, so collecting it pinned the param at :any forever — the type a
caller supplied (a vector of records) was lost, and the fn's field reads stayed
generic.
Skip a same-position pass-through arg in the self-recursion collection (contribute
the join identity). It can't add information — param i ⊇ param i is trivial — so
dropping it is sound; the param is still constrained by every external caller and
by any non-pass-through recursive arg. Applies to both self-recursion paths: a
`defn` recursing through its var, and a named fn literal recursing via its
self-local.
This is why ray-cast's `ray` typed (its recursion passes a fresh ray) but
`hittables` didn't (passed through). With the fix, hittables keeps its
vec<Sphere> element type, so hit-all's reduce element — and hit-sphere's reads —
type without any hint: ray tracer 38.4s -> 31.3s (~1.23x) with no annotations.
run-wp.ss: a recursive fn threading a vec param through keeps its element type.
make test / shakesmoke green, selfhost holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
jolt.host/record-ctor-key and record-type? were stubs returning nil since the
rehost, so phint-of always produced nil — a ^Record param hint (^Sphere s) was
silently dead. The inference only ever typed a record param when its callers
passed a concrete ctor (whole-program), so a fn called with an untyped value (a
vector element, a value threaded through recursion) read its fields generically.
Resolve the tag against the record registry (records.ss): chez-find-ctor-key maps
a ^Type tag to the ctor-key "ns/->Name" the inference seeds with, preferring the
compile ns and falling back to any registered record of that simple name (cross-ns
/ imported). record-type? / record-ctor-key call it. Runtime .ss — no re-mint (the
analyzer reaches record-ctor-key through the host var).
With this, a ^Sphere/^Ray-hinted hit-sphere in the ray tracer types its params, so
its 48 generic jolt-get field reads + boxed arithmetic become bare-index reads +
fl-ops: ray tracer 38.4s -> 23.9s (~1.6x), make test / shakesmoke green, selfhost
holds, 0 new divergences. A wrong hint stays safe — jrec-field-at falls back to
jolt-get for a non-record.
run-fieldnum.ss: a ^V-hinted param (no inferable caller) bare-indexes + unboxes,
with no generic jolt-get left.
Co-authored-by: Yogthos <yogthos@gmail.com>
A record field tagged ^double now reads back as a flonum and feeds the numeric
pass, so hintless arithmetic over those fields lowers to fl-ops — the leaf-numeric
analog of the ^Vec3 nested-field hints. Combined with the whole-program :double
param inference, a vec3-dot over a ^double-fielded record unboxes end to end with
no per-fn hints.
records.ss: a ^double field tag passes through resolution, and the ctor (and a
mutable-field set!) coerce a ^double field to a flonum — JVM primitive-field
parity (jolt returned an exact 1, not 1.0, before), and what makes reading the
field back as :double sound for an fl-op.
types.clj: field-type-from-tag maps "double" -> :double, and a keyword/get lookup
whose result is :double annotates the node :num-read :double. numeric.clj reads
that annotation and classifies the field read as a :double operand, so the
enclosing arithmetic specializes — the read itself keeps its jrec-field-at/jolt-get
emit.
run-fieldnum.ss gate: ctor coercion (int field -> flonum), field-field arithmetic
emitting fl*/fl+, and an untagged field staying generic. make test / shakesmoke
green, selfhost holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
The closed-world fixpoint (#226) flowed record types across fn boundaries; this
adds a numeric refinement so a hintless fn whose every call site passes a flonum
has its param unboxed to fl-ops, no ^double hint needed.
Lattice gains :double, a flonum refinement of :num: two doubles join to :double,
a double joined with anything else widens to :num — so a param is :double only
when every contributing value is a flonum, which is what makes the fl-op sound.
infer types a flonum literal and flonum arithmetic (+ - * / min max inc dec over
double/int-literal operands) as :double, and the fixpoint joins those across call
sites and return types like any other lattice value.
The bridge to the existing hint-directed pass is a synthetic [param :double]
nhint: wp-infer! stashes the :double params separately from the structural seeds,
and run-passes injects them as nhints before numeric/annotate, so the fl-op
emission and the exact->inexact entry coercion (a no-op on a proven flonum) apply
unchanged.
Sound subset only: :double, never :long — an untyped integer can be a bignum and
fx-ops would overflow/diverge from jolt's arbitrary precision. So an integer
caller leaves a param generic; an escaped fn (unknown callers) keeps :any.
run-numwp.ss gate: cross-fn :double propagation incl. through a flonum-returning
helper, the integer-caller and escape negatives, and the full run-passes path
emitting fl* + entry coercion. make test / shakesmoke green, selfhost holds, 0
new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
The devirtualized protocol call emitted find-protocol-method on the inferred
record tag, but a record can satisfy a protocol via an Object/host-tag default
rather than a direct impl — find-protocol-method on its own tag misses that,
while protocol-resolve walks to the default. So a record relying on
(extend-protocol P Object ...) resolved under ordinary dispatch but applied #f
under devirt and crashed. Closed-world opt builds only; the gate previously
covered just direct inline/extend-type impls so it shipped green.
Emit devirt-resolve, which tries the static tag and falls back to
protocol-resolve on a miss — same fast path, correct regardless of how the
record satisfies the protocol. Mirrors jrec-field-at falling back to jolt-get.
The receiver binds to one temp so it feeds the resolve and the application
without double-evaluating a side-effecting arg 0.
Also widen the whole-program fixpoint to :any on hitting the iteration cap: a
non-converged pre-fixpoint is more specific than the least fixpoint, so seeding
it would be unsound. Not reached in practice (~2 passes); a defensive floor.
run-devirt.ss gains an Object-default case. make test / shakesmoke green,
selfhost holds, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the inference types a keyword-lookup receiver as a record — it carries the
field-order :shape and :hint :struct from the whole-program fixpoint — the back
end reads the field by its declared slot via jrec-field-at instead of jolt-get.
That skips the jolt-get case-lambda, the dispatch fn, and the field-key
hashtable lookup, leaving a jrec? check + a static-index vector-ref.
jrec-field-at falls back to jolt-get when the receiver isn't the expected record
(a map downgraded by dissoc, or a value the inference mistyped), so it stays
correct if the static type is wrong. Only the no-default form takes the bare
path (a declared field is always present).
Sound only for non-nil records: a self-recursive param that can be nil (e.g.
binary-trees check-tree, whose untagged child is nil at leaves) types :any and
keeps jolt-get — the whole-program fixpoint demotes it. The target is non-nil
record params, like a Vec3 dot product (~5% there; boxed-flonum arithmetic
dominates the rest, a separate numeric lever).
run-fieldread.ss gate: emitted form uses jrec-field-at at the right slot and
matches jolt-get for each declared field; a non-field key and a default-arg form
keep the generic path. make test / shakesmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the inference proves a protocol call's receiver is one record type, the
back end resolves the impl by that static tag (find-protocol-method) instead of
routing through the protocol var -> jolt-invoke -> protocol-resolve, which
re-derives the tag and walks the type table. Same table lookup, minus the
var-deref, the rest-cons, and the receiver-type computation.
Fires only on a monomorphic site: a megamorphic receiver joins to :any and
carries no :devirt-type, so it keeps ordinary dispatch (the dispatch bench is
unaffected). The annotation comes from the whole-program fixpoint typing a
reduce/HOF element or a ctor return as a specific record.
Modest on the dispatch benchmarks (~6% on mono-dispatch) — float boxing in the
reduce accumulator dominates there, a separate numeric lever — but it removes
the dispatch overhead wherever a typed receiver is known.
run-devirt.ss gate: emitted form uses find-protocol-method, and evaluating it
matches ordinary dispatch for an inline impl, an extend-type impl, and the
non-devirt path. make test / shakesmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
Re-derive each app fn's param types from its call sites under --opt, so a
record type flows across fn boundaries: a ctor's return reaches a callee
param, and a typed vector's element reaches a HOF closure's param. The back
end can then bare-index field reads and devirtualize protocol calls at those
sites (it reads the resulting :hint/:devirt annotations; consuming them is
separate work).
This rebuilds the inter-procedural driver the Janet host had — the API
(infer-body/reinfer-def) survived the rehost but nothing drove it, and the
record-shapes/protocol-methods registries were empty stubs.
- records.ss: populate record-shapes (ctor key -> fields/tags/type, resolving
nested record field tags) and protocol-methods (method var -> [proto method])
registries at deftype/defprotocol load time; jolt.host accessors materialize
them.
- passes/types.clj: wp-infer! runs a closed-world fixpoint joining call-site
arg types into callee params; reinfer-def re-seeds each def at emit. Self-
recursive calls and fn-level recur are collected so a recursive fn's params
are constrained by its recursion, not just external callers — else a param
the recursion widens (e.g. binary-trees check-tree, whose untagged child can
be nil) would be unsoundly typed non-nil. A fn used in value position keeps
:any params (callers unknown). Megamorphic sites join to :any.
- build.ss: analyze all app forms and run the fixpoint before per-form emit.
- run-wp.ss: gate (cross-fn propagation, escape soundness, self-recursion).
make test / shakesmoke green, 0 new divergences, selfhost holds.
Co-authored-by: Yogthos <yogthos@gmail.com>
The persistent vector was a flat Scheme vector with copy-on-write: every conj
copied the whole backing array, so building an n-element vector was O(n^2). On the
collections bench that's vec-sum building 7500 elements with 227MB of copies, 90%
of the bench's allocation.
Replace it with Clojure's PersistentVector — a 32-way trie plus a trailing tail
chunk. conj appends to the tail and, when it fills, path-copies it into the trie,
so conj is O(1) amortized and a linear build is O(n). nth/assoc/pop are
O(log32 n). make-pvec (build a trie from a flat vector) and pvec-v (materialize
back) stay as compatibility shims, so the ~14 callers that read the backing array
— all one-shot conversions in =, hash, seq, meta-copy, transients, the reader —
are untouched; only this file's internals change.
vec-sum 70ms/227MB -> 1ms/3MB; collections 10.4x -> 4.0x vs JVM, under the 5x
target. 5 new corpus rows plus boundary stress (level transitions at 32 and 1024,
pop-collapse, assoc at every index) cover the trie.
Co-authored-by: Yogthos <yogthos@gmail.com>
A loop test like (or (>= i cap) (> ... 4.0)) desugars to
(let* [g (>= i cap)] (if (truthy? g) g (> ... 4.0))) and the whole thing was
wrapped in jolt-truthy? because returns-scheme-bool? only looked at :const and
:invoke nodes, not the let*/if an or/and expands to. The wrapper defeats Chez's
branch inlining on the hot loop edge.
Make returns-scheme-bool? recursive over :if (both branches bool), :let (body
bool, tracking which bound locals hold a Scheme boolean), and :local (in that
set). or/and over bool-returning ops then read as Scheme booleans and the outer
wrapper drops. Still sound: eliding only when the value is provably #t/#f — a
jolt-nil is a truthy record in Chez, so a false positive would be a real bug, and
the recursion only proves bool-ness through ops already known to return one.
No bench regression; the win lands on hinted float loops where the branch, not
boxed arithmetic, is the cost.
Co-authored-by: Yogthos <yogthos@gmail.com>
defprotocol emitted one variadic (fn [this & rest] (protocol-dispatch P m this
(list->cseq rest))) per method, so every protocol call — even a no-extra-arg one
like (area s) — consed a rest list, wrapped it in a cseq, var-deref'd
protocol-dispatch, and jolt-invoke'd it (consing again). On mono-dispatch that was
2.07GB of allocation, ~65% of the benchmark.
Emit one fixed-arity clause per declared arglist instead. The 1/2/3-param arities
call positional protocol-dispatch{1,2,3}, which resolve the impl (by record tag,
reify method, or host-tag extension — factored into protocol-resolve) and apply it
directly; no rest-list, no seq round-trip. The dispatchN entry points are in the
native-op table so the shim calls bind straight to the records.ss procedures
rather than var-deref. 4+ params fall back to the variadic protocol-dispatch.
mono-dispatch 1.5s/2.07GB -> 0.69s/280MB; dispatch 26x -> 12.2x, mono-dispatch
111x -> 51x vs JVM. 5 new corpus rows pin multi-arity methods, host-type args,
and protocol-method-as-value against JVM Clojure.
Co-authored-by: Yogthos <yogthos@gmail.com>
Records were a jrec holding an alist of (kw . val) conses: ~113B/node, built
fresh per construction, field reads a list scan. Replace that with a shared
per-type descriptor (tag + field keywords + an eq?-keyed keyword->index table)
plus a flat per-instance value vector and an extension map for any non-field
keys assoc'd on (jolt-nil when there are none). Construction now allocates one
vector instead of a cons chain and a field read is an index lookup. binary-trees
construction allocation drops 2.085GB -> 1.19GB.
That alone barely moved binary-trees wall-time: profiling showed the read loop,
not allocation, dominates, and the read loop's own allocation came from (nil? l)
lowering to (jolt-invoke (var-deref "clojure.core" "nil?") l), which conses its
args every call. Add nil?/some? to the backend native-op table so they inline to
jolt-nil?/jolt-some? (and drop the truthy wrapper, like the other predicates).
check-tree's read loop goes from 1.476GB allocated to zero; binary-trees 18.9x
-> 9.7x vs JVM. The remaining gap is the field-read dispatch chain (jolt-c3mw).
Two JVM divergences fixed along the way, both certified:
- dissoc of a declared field downgrades a record to a plain map (was kept as a
record); an extension key still drops cleanly.
- map->R keeps extension keys (was dropping anything outside the declared basis).
16 new corpus rows pin assoc/dissoc/count/keys/seq/=/hash/extension-field
behavior against JVM Clojure.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Make the benchmark harness build optimized binaries on Chez
bench/run.sh was Janet-era: it invoked a 'jolt' binary and set
JOLT_DIRECT_LINK/JOLT_WHOLE_PROGRAM, none of which exist on Chez, where
'joltc run -m' runs fully unoptimized (direct-link and inline default off). So
the suite was measuring jolt's unoptimized path.
run.sh now compiles each benchmark to an optimized AOT binary (joltc build
--direct-link --opt) and times it against JVM Clojure on the same portable
source, auto-detecting the Chez kernel dev files like build-smoke.sh. Adds
bench/deps.edn so joltc resolves the namespaces, NO_JVM to skip the reference.
mandelbrot.clj dropped its jolt.png require so the JVM reference can run it; the
picture demo moved to mandelbrot_png.clj (jolt-only). README scorecard refreshed
with current Chez numbers and the two-regime read (compute ~8-10x substrate floor;
dispatch/alloc ~120-330x architectural gaps the passes don't touch). Stale
'jolt -m' header lines point at bench/run.sh.
* Emit direct self-calls for named-fn self-recursion
A self-recursive call to a named fn compiled to (jolt-invoke fib ...) instead of
a direct (fib ...): emit-invoke handled a :local callee only when it was NOT a
known proc, so a :local that IS in *known-procs* (the letrec-bound self-name) fell
through to the :else jolt-invoke branch. Now a :local known proc emits a direct
Scheme call — no jolt-invoke, no per-call arg-list consing; case-lambda handles
arity.
fib 30: 63.3ms -> 4.7ms (faster than JVM Clojure's 7.1ms; was 9x slower). The win
is on every self-recursive non-loop fn, including the compiler's own. No semantic
change — selfhost holds, make test green, shakesmoke/buildsmoke byte-identical.
Re-mint (backend is seed). Corpus rows pin self-recursion across fixed/multi/
variadic arities.
* Intern no-ns keywords without per-call allocation
(keyword #f name) built a fresh combined-key string (string-append) on every
call just to do the intern-table lookup — ~80 bytes of garbage per (:kw x), map
literal, keyword arg, etc. A no-ns keyword now interns in a table keyed by the
name string directly, so a lookup of an already-interned keyword is one
hashtable-ref with no allocation. The ns table keeps the combined key; both share
the keyword-t khash (equal-hash of the combined key) so hash values are unchanged.
Small time win on its own (the field-read dispatch dominates hot record code —
see jolt-unx4) but removes per-call keyword allocation everywhere. Runtime .ss,
no re-mint; identity/=/hash unchanged, make test green.
* Fast record field reads: single eq? scan, skip the get-arm walk
(:field rec) / (get rec :field) lowers to (jolt-get rec kw), which walked the
get-arm list to reach the jrec arm, then did jrec-has? + jrec-lookup — TWO linear
scans, each comparing keys through the generic jolt=2 equality dispatcher. Field
keys are interned keywords, so:
- jrec-key=? compares a keyword query by eq? (jolt=2 only for non-keyword keys),
- jrec-ref does ONE scan (vs has?+lookup) and runs a deftype's ILookup valAt only
when the field is genuinely absent (present-nil still returns nil, not default),
- jolt-get-dispatch checks jrec? first, skipping the get-arm walk for the hottest
get target. jrec-lookup/jrec-has? (used by =, contains?, etc.) get the fast
compare too.
binary-trees 135x->18.9x, dispatch 121x->26.4x, mono-dispatch 327x->108x vs JVM.
Runtime .ss (collections.ss + records.ss), no re-mint; make test + shakesmoke +
buildsmoke green, record get/assoc/keys/=/count semantics unchanged.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
bench/run.sh was Janet-era: it invoked a 'jolt' binary and set
JOLT_DIRECT_LINK/JOLT_WHOLE_PROGRAM, none of which exist on Chez, where
'joltc run -m' runs fully unoptimized (direct-link and inline default off). So
the suite was measuring jolt's unoptimized path.
run.sh now compiles each benchmark to an optimized AOT binary (joltc build
--direct-link --opt) and times it against JVM Clojure on the same portable
source, auto-detecting the Chez kernel dev files like build-smoke.sh. Adds
bench/deps.edn so joltc resolves the namespaces, NO_JVM to skip the reference.
mandelbrot.clj dropped its jolt.png require so the JVM reference can run it; the
picture demo moved to mandelbrot_png.clj (jolt-only). README scorecard refreshed
with current Chez numbers and the two-regime read (compute ~8-10x substrate floor;
dispatch/alloc ~120-330x architectural gaps the passes don't touch). Stale
'jolt -m' header lines point at bench/run.sh.
Co-authored-by: Yogthos <yogthos@gmail.com>
A type-aware audit (~190 collection expressions vs reference Clojure) found four
divergences the corpus missed — value-equality (= [0 1] '(0 1)) hides type and
laziness differences. Fixed, with type-predicate + over-infinite corpus rows that
pin them.
- partition-all [n coll] built vector chunks; JVM chunks are seqs. (The [n step
coll] arity was already correct, as is the partition-all transducer, whose
chunks are vectors in JVM too.) Now builds seq chunks.
- replace always returned a vector (mapv) and was eager; JVM is type-preserving —
a vector maps to a vector, any other seqable to a lazy seq.
- sequence eagerly realized its source (into-xform), so (first (sequence (map inc)
(range))) hung. Rewrote as a transformer iterator: pull one input at a time,
buffer the step outputs, emit lazily, run the completion to flush a stateful
xform. eduction builds on it (lazy, no longer an eager vector).
- mapcat and (apply concat coll-of-colls) hung over an infinite source because
jolt-apply seq->lists the trailing arg and mapcat seq->lists the map result.
Added lazy-concat-seq (lazily flatten a seq of colls); mapcat uses it directly,
and apply special-cases concat (its result is lazy) to route through it.
Docs: a cross-cutting return-type + laziness contract in docs/spec/09-core-library;
SPEC.md notes that = masks type/laziness so they need predicate / over-infinite
rows. EBNF is reader syntax only — unaffected.
Seed change (partition-all/replace/eduction are clojure.core overlay) -> re-mint;
selfhost holds. make test + shakesmoke + buildsmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Reader records source line/column on list forms
The reader stamps 1-based :line/:column metadata on every list form (plus
:file when load-jolt-file is reading a file), and jolt.host/form-position
reads it back so the analyzer's :pos scaffold finally gets real data. A
left-to-right cursor counts newlines over the delta between successive forms,
so it stays O(n). Vector/map/set literals are untouched (their metadata is a
runtime value the analyzer would have to wrap in with-meta); empty () can't
carry meta. ^meta now merges onto the position keys instead of clobbering them.
Re-mint is byte-identical (the backend doesn't emit :pos), so this is a pure
scaffold for the error-location work that follows.
* Report source location on uncaught errors
Each top-level form records its source position (thread-local) before it
compiles+evals, and cli.ss jolt-report-uncaught appends 'at file:line:col'
when an error propagates out. Covers joltc -e, joltc run <file>, and
load-string — every interpreted path. Top-level granularity, one set per
form; deeper frames come from the Phase 2 frame walk.
Runtime .ss only, no re-mint.
* Clojure stack traces via source registry + native frame walk
A direct-link build emits (jolt-register-source! short-name ns name file line)
once per fn def — at definition time, so zero per-call cost. On an uncaught
error the reporter walks Chez's native continuation frames (jolt-throw captures
the live continuation via call/cc; host conditions carry their own
&continuation), maps each frame's procedure name through the registry, and
prints a Clojure backtrace 'ns/name (file:line)'. Wired into both the cli and a
built binary's launcher.
Frames are keyed by the short munged fn name Chez actually reports (emit-fn's
letrec self-binding), not jv$ns$name; a cross-namespace collision degrades to
the bare frame name rather than a wrong attribution. The analyzer carries the
original form's position through defn macroexpansion onto the def node.
Calling a non-fn now throws a catchable ClassCastException (via jolt-throw)
naming the operator, instead of a raw Chez error.
Caveats (documented in source-registry.ss): names map only in direct-link/AOT
closed-world builds — the open-world -e/repl/run path falls back to the
top-level location; and pervasive TCO erases tail-call frames, so a mapped
trace shows only the non-tail spine. JOLT_DEBUG_FRAMES dumps raw frame names.
Re-mint (analyzer + backend); prelude byte-identical (direct-link off during
mint). Corpus rows certified, build-smoke asserts the trace.
* Propagate source position through macroexpansion
hc-expand-1 now carries the macro call form's :line/:column onto the top of a
list expansion that has none of its own (merged under any meta the macro set),
so errors and stack traces in macro-generated code point at the call site —
Clojure parity. The analyze recursion re-expands inner macros, so each level's
top form picks it up, matching the reference compiler. (meta (macroexpand-1
'(when x y))) now reports the call-site line.
A direct-link fn defined through a user macro (build-app's defguarded) registers
with a real line, so build-smoke's trace assertion covers macro-defined fns.
Runtime .ss (host-contract.ss) — no re-mint; selfhost holds.
Phase 3's optional items are deferred: :line-in-ex-data has no clean consumer
(it would pollute ex-data, break = and printing, and positions already surface
via the trace + top-level location), and Chez source-object emission is a large
backend change the jv$-name registry already sidesteps.
* Review fixes: registration key, thread-locals, debug flag timing
- Register a fn under the name Chez actually reports for its frame, not the def
name: a named fn literal whose name differs from the def (def foo (fn bar …))
is framed as 'bar', and an anonymous fn def (def foo (fn …)) as jv$ns$foo.
Both previously registered under the def name and so never appeared in traces.
- rdr-source-file / rdr-pos-cursor are thread parameters, so concurrent compiles
(futures, core.async) don't clobber each other's file/line attribution.
- Read JOLT_DEBUG_FRAMES at call time: a built binary evaluates top-level forms
at heap-build time, where a load-time getenv is always unset.
Re-mint (backend + reader); prelude byte-identical, selfhost holds.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Review found (< 1M 2M) worked but (min 1M 2M) threw — incoherent. Wire min/max
the same way as the other ops: value-position jolt-min/jolt-max shims (new in
seq.ss, added to core-value-procs) and call-position via bd-spec/bd-ops ->
jbd-min/jbd-max.
min/max return the original operand by value, not a coerced copy, matching
Clojure: (min 1M 2.0) -> 1M, (max 1M 2.0) -> 2.0, (min 1.50M 2M) -> 1.50M; a tie
keeps the second operand ((max 1.5M 1.50M) -> 1.50M). bigdec mixed with a flonum
in call position stays in the documented :any/contagion gap (value position
handles it). Re-mint; 6 more JVM-certified rows.
A direct (+ 1.5M 2.5M) emits a raw Chez + that rejects the bigdec record. Rather
than guard every arithmetic call site (measured 2-4x on unhinted fixnum loops),
let the analyzer dispatch where it can prove the type.
jolt.passes.numeric seeds a :bigdec kind from the M-literal and flows it through
let/loop/if like the existing :double/:long kinds; an arithmetic/comparison invoke
whose operands are all bigdec (integer literals allowed) gets :num-kind :bigdec.
The back end (bd-ops + emit-numeric) lowers those to the bigdec.ss engine
(jbd-add/-sub/-mul/-div, jbd-lt?/…, jbd-zero?/-pos?/-neg?, jbd-quot/-rem).
Zero cost on non-bigdec code: with no bigdec literals present the kind never
arises, so emission is byte-identical — the re-mint leaves prelude.ss unchanged,
only image.ss (the compiler) moves. Gaps (filed): a bigdec mixed with a flonum in
call position, and a bigdec the analyzer types :any, still hit the raw op and
throw; use value position or a literal-typed let.
Re-mint (numeric/backend are seed sources). 16 JVM-certified corpus rows.
bigdec values existed but +,-,*,/ and compare threw — the header even said
"arithmetic contagion is not modelled". Add the scale-aware engine on the
{unscaled, scale} pair (jbd-add/-sub/-mul/-div + comparison helpers) following
java.math.BigDecimal's rules: add/sub align to the larger scale, multiply adds
scales, divide gives the exact quotient at minimal scale or throws
ArithmeticException on a non-terminating expansion. Clojure contagion: a bigdec
mixed with an integer stays bigdec, a flonum operand wins (result is a double).
Wire it into the value-position shims only — jolt-add/-sub/-mul/-div (what
(reduce + bigs)/(apply * bigs) lower to) and compare — so the inlined native hot
path is untouched. A call-position (+ 1.5M 2.5M) still reaches the raw Chez op;
that needs the analyzer's :bigdec type (next).
Runtime .ss only, no re-mint. 13 JVM-certified corpus rows.
The host/chez directory mixed jolt's own runtime (value model, seq, reader,
vars, ns, multimethods) with the shims that emulate the JVM: java.* / javax.*
classes, clojure.lang interfaces, and the host-class registry they hang off.
Move that JVM-emulation layer into host/chez/java/ so it reads as a distinct
unit instead of being interleaved with the platform runtime.
Moved (content unchanged): host-static, host-static-methods,
host-static-classes, host-class, dot-forms, records-interop, byte-buffer,
io, io-streams, inst-time, java-time, bigdec, natives-queue, natives-str,
natives-array, math, concurrency, async, ffi.
The load paths in rt.ss/cli.ss and the build.ss runtime manifest are updated
to point at java/; the build inliner follows the (load ...) strings, so the
AOT path needs no other change. All runtime shims, no seed source touched
(the three .clj edits are doc comments), so no re-mint.
Gate green: make test (selfhost fixpoint, certify 0-new, sci 211, infer),
shakesmoke (4 apps byte-identical).
A post-conformance review (chiasmus) flagged fresh-sym defined byte-identically
in 00-syntax and 30-macros; 00-syntax loads first, so the second is redundant.
Also note why deftype uses group-by-head while extend-protocol uses
parse-extend-impls (the latter must treat a computed class type in head position).
No behavior change.
Conformance gaps surfaced re-running the library suites:
- defn now keeps a leading docstring as :doc metadata — it was dropped, so
(:doc (meta #'f)) was always nil. Rides the def docstring slot.
- assert (and :pre/:post) throw a real AssertionError instead of an ex-info, so
(catch AssertionError …) / (thrown? AssertionError …) match, with Clojure's
"Assert failed: <msg>\n<form>" message.
- instance? clojure.lang.Seqable was conflated with ISeq, so a vector/map read
as not-Seqable. Split them: Seqable covers every persistent collection, ISeq
only seqs.
Running cognitect aws-api's pure test namespaces (signing/shapes/protocols/
util/retry/endpoints) surfaced general gaps:
- extend-protocol/extend-type accept a computed class type, e.g.
(Class/forName "[B") for the byte-array class — the byte-array idiom data.json
and aws-api use. The macro grouping handled only symbol/nil heads (it crashed on
a list type); type->name resolves a Class value via .getName; a byte-array
dispatches on the "[B" host tag.
- java.nio.ByteBuffer over a jolt byte-array (wrap/allocate/get/put/array/
remaining/position/limit/duplicate/flip), plus extend-protocol to it.
- java.util.Arrays (equals/copyOf/copyOfRange/fill) and java.util.Random
(nextBytes/nextInt/…).
- java.net.URI/create and clojure.lang.RT/baseLoader statics.
- clojure.core.async/promise-chan (deliver-once, peek-don't-pop).
- a failed java.time parse throws DateTimeParseException (typed), so
(catch DateTimeParseException …) matches it instead of leaking an untyped
condition.
The XML side lives in the jolt-lang/xml library (libxml2 over jolt.ffi); ByteBuffer
stays in core as a generic java.nio primitive.
Gate: make test green (corpus +6 JVM-certified rows, 0 NEW divergence; unit
553/553; SCI 211).
Shaking out clojure.core.memoize (207 assertions, 0 fail) cleared several
general gaps:
- deref/@ on a deftype or reify implementing clojure.lang.IDeref dispatches to
its deref method (RetryingDelay / make-derefable).
- deftype mutable fields (^:unsynchronized-mutable / ^:volatile-mutable) are
read live: a set! within a method is observed by a later read in the same
invocation, not the entry-time capture. Needed for double-checked locking.
Immutable fields stay let-bound. Field reads rewrite to (.-field inst) with
lexical-shadow tracking.
- def metadata values are evaluated, like Clojure: ^{:k (f)} stores (f)'s
result and ^{:af some-fn} the fn. :tag stays a literal hint.
- try dispatches catch clauses by class in order via the exception supertype
hierarchy; a non-matching value re-throws, an untyped host condition is caught
by a RuntimeException/Exception/Throwable clause. Previously the last clause
won and the class was ignored.
- locking takes a real per-object monitor (recursive Chez mutex) now that
futures/agents/threads share one heap; it was a no-op.
- supers/ancestors reflect a small modeled JVM interface hierarchy, so
(ancestors (class f)) yields Runnable/Callable (core.memoize's arg check).
- AssertionError / Error constructors.
JOLT_FEATURES is gone from the docs: it isn't read anywhere on Chez, and the
reader already includes :clj in its default feature set. RFC 0002's
{:jolt :default} design was reverted in the reader; docs now match the code.
Raises the SCI floor 205 -> 210.
Replace the strong-ref stub with genuine reclamation. The referent is held
through a weak-cons, so Chez's generational collector reclaims it once it is
otherwise unreachable (the pair's car becomes the bwp object, and .get returns
nil). A guardian registered on the referent makes the reference itself available
the instant its referent is collected, which ReferenceQueue.poll surfaces as
enqueued — the same hook clojure.core.cache's clear-soft-cache! drains.
Chez has no softer-than-weak reference, so a SoftReference clears on
unreachability rather than under memory pressure: a SoftCache evicts more eagerly
than the JVM's but is now real GC eviction, not an unbounded strong cache.
WeakReference gets the same (faithful) semantics. Added System/gc -> a full
collect so callers (and the queue) can force the cycle.
core.cache stays 1314/0/0 (its test values are immortal literals). Corpus row for
System/gc; make test + shakesmoke green.
Closes the last clojure.core.cache gaps (now 1314/0/0, including the 1000-thread
cache-stampede):
- java.lang.Thread over Chez fork-thread (shared heap): (Thread. thunk) + start/
join/run/isAlive, on a "user-thread" tag distinct from Thread/currentThread's
interrupt shim. java.util.concurrent.CountDownLatch (count/mutex/condition).
- java.util.concurrent.ConcurrentHashMap = the mutable HashMap shim; get / count /
contains? read it (clojure.core), which SoftCache uses on its backing map.
- java.lang.ref.SoftReference / ReferenceQueue: no JVM GC reference semantics, so
the referent is held strongly (a SoftCache is unbounded rather than GC-evicting),
but enqueue / poll work so clear-soft-cache! drains the queue.
JVM-certified corpus rows. make test + shakesmoke green.
Further clojure.core.cache fixes (198 -> 257 of its assertions):
- delay: a throwing body re-ran on every force and never became realized?. Run it
once like Clojure's Delay — cache the exception, mark realized, re-throw the same
on each deref. Fixes value-fn memoization / cache-stampede protection.
- deftype/defrecord: a method name appearing in two protocols with different
arities (data.priority-map's seq is in IPersistentMap [this] AND Sorted
[this asc]) registered per-protocol and shadowed; merge clauses by name across
all protocols into one multi-arity fn.
- empty?/peek/pop (IPersistentStack) dispatch through a deftype's methods; (= a-
deftype other) uses its equiv method (so caches compare to their backing map);
seq handles a host iterator (iterator-seq over .iterator).
- pop of an empty PersistentQueue returns it, like the JVM (was an error).
JVM-certified corpus rows. make test + shakesmoke green.
General fixes shaken out running clojure.core.cache (66 -> 198 of its assertions):
- Map destructuring applied an :or default only for :keys/:strs/:syms, not a
direct {x :x} binding — so {x :x :or {x 9}} (and the & {…} kwargs form) ignored
the default. Apply it for the direct binding too.
- fn didn't implement :pre/:post: a leading conditions map was evaluated as a body
literal (so % was unbound and (.q %) blew up). Recognize it and assert pre
before the body, bind % to the result, assert post, return %.
- (.q inst) on a deftype field with no matching method reads the field, like the
JVM (was "No method q").
- A deftype implementing the clojure.lang collection interfaces now dispatches
dissoc (without), contains? (containsKey), peek/pop (IPersistentStack), and
keys/vals (via its Seqable seq) through its methods — they were field-only, so
core.cache's caches and data.priority-map didn't behave as maps.
JVM-certified corpus rows for each. make test + shakesmoke green.
input-bytes handled in-stream/bytevector/byte-array sources but not a host
byte-input-stream table (:jolt/input-stream — http-client's ByteArrayInputStream),
so io/copy fell through to rendering it as text (#[chez-htable …]) instead of its
bytes. Drain it like slurp does. Fixes http-client's response-body handling
(jolt-bjbi): its suite goes 100/16-fail -> 116/0, and the ring adapter's.
io/copy handled file/stream/writer targets but not a host byte-output-stream
table (jolt-lang/http-client's ByteArrayOutputStream shim, :jolt/output-stream),
erroring 'don't know how to write to'. Dispatch through the shim's .write method,
byte-exact for a byte source — the JVM's io/copy writes to any OutputStream.
Shaken out getting ring-defaults (and its ring-core/anti-forgery/session stack)
to load and serve static resources on jolt. All general fixes, all runtime:
- Class/forName throws a catchable ClassNotFoundException for a class jolt can't
back (it returned a broken truthy value for any name, and crashed on use). Lets
the common (try (Class/forName "optional.Dep") (catch ...)) probe libraries use
to detect an absent dependency work — e.g. ring's joda-time check.
- deps: reconcile native libs (and source roots) in one step, deduped by library
identity, instead of the ad-hoc distinct at each call site. An app pulling two
libs that declare the same shared object (libcrypto via both jolt-crypto and
http-client) now includes and loads it once.
- io: a File answers getProtocol ("file") / getFile so resource-serving
middleware that expects io/resource to hand back a file: URL works; the
classloader gains getResources (every source root holding the resource).
- clojure.string/replace accepts a char match/replacement, like the JVM.
JVM-certified corpus rows for the Class/forName and string/replace behavior.
Finishes core.match — its full test suite (115/115) now passes, including the
two patterns the earlier work left out:
- Regex-literal patterns. A #"…" now reads as a regex VALUE (Clojure parity: the
reader constructs the Pattern, so a macro receives a regex, not jolt's tagged
form), and the analyzer compiles a regex value to the same :regex IR leaf via
its source. emit-quoted handles a quoted regex; a regex value carries the
java.util.regex.Pattern host tag so extend-protocol/instance? dispatch on it.
- Primitive-array patterns. A ^Type hint's :tag is now the SYMBOL (e.g. `ints`),
matching the JVM, so core.match's array-tag lookup engages the array
specialization (alength/aget). jolt's :tag consumers already tolerate a symbol
(hc-cell-num-ret normalizes; tag->nkind/def-meta handle both).
Also: a library-conformance directive in CLAUDE.md, and the supported-libraries
list (docs + site) simplified to one-line entries — a listed library is assumed
to work fully, so no tallies or feature enumerations. core.match + transit-jolt
added to the list.
Seed change (reader/backend/30-macros) -> re-minted; the rest runtime. JVM-
certified corpus rows; the stale `symbol hint -> :tag` divergence is dropped from
the allowlist (jolt now matches the JVM). make test + shakesmoke green.
Running clojure.core.match (a macro-heavy library that builds its compiler out
of deftypes implementing clojure.lang interfaces) shook out a cluster of general
gaps. Its own suite goes from not-loading to 111/115 assertions.
- deftype/defrecord implementing a clojure.lang collection interface now drives
the core fns: Indexed -> nth, Counted -> count, Associative -> assoc, ILookup
-> get/valAt (non-field keys only, so a method's own field bindings don't
recurse), ISeq -> seq/first/rest, IPersistentCollection -> conj, IFn -> the
value is callable. A jrec is still a map of fields by default; the interface
method wins when declared.
- Multi-arity inline methods are grouped into one fn (a type with (nth [_ i]) and
(nth [_ i x]) kept only the last before). Built as data, not a nested
syntax-quote, so a `(= ~ocr ~l) method body keeps its unquotes.
- instance?/satisfies? recognize a protocol a type implements, including a MARKER
protocol with no methods (core.match's IPseudoPattern) — deftype/defrecord now
record protocol satisfaction even with zero methods. Added ILookup/Indexed/
Counted to the instance? taxonomy for the built-in collections.
- Syntax-quote: a fully-qualified class name (clojure.lang.ILookup) stays bare
instead of being namespace-qualified; (unquote x) is detected in a lazy seq
(a macro that builds its template with map, e.g. deftype's rewrite-set).
- clojure.set union/intersection/difference are variadic (& sets) + union 0-arity.
- java map view methods: keySet/values/entrySet/size/isEmpty.
- deprecated java.util.Date getters (getYear/getMonth/...) + the multi-arg
(Date. year-1900 month0 date hrs min) constructor.
Seed change (deftype/defrecord macros + clojure.set) -> re-minted; the rest are
runtime. 11 JVM-certified corpus rows; make test + shakesmoke green.
clojure.edn's __read-tagged seam called (empty-pmap) — applying the empty-pmap
VALUE as a procedure — so an unknown tag (e.g. #object[...] from a JVM-printed
object, or any unregistered #foo) crashed the Chez VM with "attempt to apply
non-procedure" and surfaced a malformed condition (class :object, nil message)
instead of a catchable error.
Throw a clean ex-info naming the tag, matching the JVM's "No reader function
for tag <tag>". A reader port over edn (transit-jolt's read-conformance skip
path, aero, etc.) now catches a real exception instead of aborting.
clojure.core/read-string stays lenient (returns the tagged form) so clojure.edn
can apply :readers / :default before falling through to this throw.
Expand java.io so libraries that touch the filesystem work unchanged.
File: the full method surface — length, lastModified, can{Read,Write,Execute},
isHidden, list, mkdir(s), delete, createNewFile, renameTo, getParentFile,
get{Absolute,Canonical}File, compareTo/equals/hashCode — plus the statics
separator / pathSeparator / createTempFile / listRoots. A File now keeps the
path as given (new File("rel").getPath() is "rel", .isAbsolute false); a
relative path resolves against JOLT_PWD only when the filesystem is touched,
matching the JVM. slurp/spit and the dir helpers go through the same
resolution, fixing a spit-vs-slurp inconsistency.
Streams (host/chez/io-streams.ss) — each a jhost wrapping a Chez port, so
buffering, EOF and binary<->char transcoding come from Chez:
- FileInputStream / FileOutputStream / ByteArrayInputStream /
ByteArrayOutputStream / BufferedInputStream / BufferedOutputStream
- FileReader / FileWriter / InputStreamReader / OutputStreamWriter /
BufferedReader / BufferedWriter
Buffered* return the wrapped stream (Chez ports are already buffered).
clojure.java.io: input-stream/output-stream now yield real byte streams (were
aliased to the char reader/writer); added copy (byte-exact for byte sources),
make-parents, delete-file. with-open also closes file-writer/port-writer/
print-writer (a pre-existing gap).
All runtime shims, no re-mint. 15 JVM-certified corpus rows; make test +
shakesmoke green.
Shaking out tick's api and alpha.interval suites (api 353->359, interval
0->103 passing) cleared a set of general gaps:
- Named-zone DST. Zones resolved to a fixed representative offset, so
America/New_York in August read -05:00 not -04:00. Add US/EU DST rules
(compact transition-date math) and make instant<->zoned, the zone rules'
getOffset, and the zoned equality arm DST-aware.
- Nanosecond zoned/offset times. Instant is nanos but atZone/atOffset/
toInstant/withZoneSameInstant and Instant/parse went through epoch-ms,
truncating sub-ms. Route them through nanos.
- Locale month/day names. A formatter dropped its Locale; carry it and add
French names so MMM under Locale/FRENCH renders "mai".
- Callable records. A defrecord implementing clojure.lang.IFn (tick's
GeneralRelation) is now invokable: jolt-invoke dispatches to its inline
invoke method. Also give collections the Iterable host tag so a protocol
extended to Iterable matches vectors/seqs.
- Imported class short names. (:import [java.time ZonedDateTime]) then
(. ZonedDateTime parse s) resolved to nil; an otherwise-unresolved bare
Capitalized name that's a registered host class now resolves as a class.
- Data readers. A project's data_readers.{clj,cljc} is loaded into
*data-readers* (reader namespaces required eagerly); registered #tag
literals in source rewrite to (reader-fn 'form). clojure.core/read-string
now applies #inst/#uuid/#"regex" and *data-readers* like Clojure.
- Duration/between accepts zoned/offset date-times.
All runtime shims, no re-mint. docs/libraries.md: tick full pass + aero.
Four general gaps, shaken out by loading clojure.spec.alpha:
- Special forms were shadowable by a same-named macro. analyze-list
macroexpanded before checking special forms, so a ns that redefs def/and/or
(spec excludes them via :refer-clojure :exclude) made a bare def resolve to
the macro instead of the special form, breaking every defn after. Now a head
in the special-form set is never macroexpanded, matching the reference
macroexpand1 isSpecial check.
- reify dropped all but the last arity of a multi-arity protocol method (spec
reifies (specize* [s]) and (specize* [s _])). The macro keyed methods by name
and overwrote; now it groups arities into one multi-arity fn.
- reify instances did not implement IObj: with-meta threw and (instance?
clojure.lang.IObj r) was false. Every Clojure reify carries metadata. with-meta
now copies the reify to a fresh identity (shared method table) and keys its
meta; instance? IObj/IMeta is true for any reify. This was the registry bug —
spec's with-name returned nil for specs, so get-spec missed.
- (set! (. Class field) val) was rejected. spec toggles
clojure.lang.RT/checkSpecAsserts this way; the analyzer now lowers it to a
jolt.host/set-static-field! call over a mutable-statics table, and a plain
Class/field read consults that table.
Also: .name/.getName on a Namespace and .ns/.sym on a Var (spec's ns-qualify /
->sym). analyzer + reify are seed sources (re-minted). spec.alpha now does
valid?/conform/cat/keys/explain-str/check-asserts. tick.alpha.interval-test still
needs time-literals data readers (separate).
The Instant jhost stored epoch-ms, so plusNanos/getNano rounded to the
millisecond and two instants a nanosecond apart compared =. Store epoch-nanos
instead: mk-instant still takes ms (scales to nanos) for the ms-based zone/Date
callers, mk-instant-nanos/inst-nanos own the nano arithmetic, and inst-ms floors
nanos back to ms. plus/minus/getNano/truncatedTo/compare/equals and the ISO
renderer all run on nanos; toString shows the fraction in 3/6/9-digit groups like
ISO_INSTANT. Fixes tick's interval coincidence test (shift end by one nano).
ZoneOffset/ZoneId (SHORT_IDS, fixed-offset + UTC + system; named zones via a
fixed-offset table), ZonedDateTime/OffsetDateTime/OffsetTime, Clock (fixed/
system, with now [clock] arity), and DateTimeFormatter integration (ofPattern
+ ISO_* constants, .format/.parse over the rich java.time values via the
inst-time.ss pattern engine). systemDefault resolves to UTC to keep the
#inst atZone/toInstant round-trip machine-tz-independent.
tick.core + tick.protocols + tick.locale-en-us load; tick's api_test runs
31 tests / 352 pass / 7 fail / 0 error. The 7 are host gaps: named-zone DST
(no tzdb), French locale month names (no locale DB), nanosecond Instant.
General fixes surfaced by tick: :ns/keys map destructuring ({:tick/keys [..]})
in 00-syntax.clj (re-minted), and extend-protocol to java.time classes
(records.ss host-type-set). 12 corpus rows certified vs JVM. make test +
shakesmoke green, selfhost holds, 0 new divergences, data.json stays 138/139.
Duration (ISO PT.. toString, between, full arithmetic), Period (between with
borrow, P.. toString, normalized), full Month/DayOfWeek enums (named constants,
print as their name — fixes the Phase-1 raw-jhost print), Year, YearMonth
(2020-02 toString, leap, atDay/atEndOfMonth), ChronoUnit (between/getDuration)
and ChronoField. The temporal machinery on the Phase-1 types now works with
ChronoUnit/ChronoField: (.plus t n DAYS), (.until t1 t2 unit), (.get/.getLong
t field), (.with t field v), (.isSupported ..), (.truncatedTo ..).
Analyzer: (. Class method args) with a class target lowers to a static call
(Class/method args) instead of mis-dispatching as an instance call on the arg
— matches JVM; needed by cljc.java-time.year. Seed re-minted; selfhost holds.
The Phase-2 cljc.java-time namespaces load; tick.core advances to a Phase-3
zone gap. 10 corpus rows certified vs JVM. make test + shakesmoke green, 0 new
divergences, data.json stays 138/139.
Core java.time value types as jolt host objects backed by the inst-time.ss
calendar engine (days-from-civil/civil-from-days/inst-fields/format-ms), in a
new host/chez/java-time.ss. tz-free reps: LocalDate=epoch-day,
LocalTime=nano-of-day, LocalDateTime=(epoch-day,nano-of-day); Instant reuses
the ms-based instant jhost (ms granularity; nano is a documented gap). Each
type registers statics (of/parse/now/MIN/MAX/...), instance methods
(plus/minus/with/get/isBefore/until/toString/...), =/hash, compare,
ISO-8601 print, instance?, and value-host-tags for protocol dispatch.
Reconciles the old ms-based local-date/local-dt stubs into the rich types
(LocalDateTime now prints ISO; .toLocalDate/.atZone paths preserved). The
four cljc.java-time namespaces (local-date/local-time/local-date-time/instant)
load. Deep temporal-field/unit machinery (range/get-long/with-field/until/
adjust-into) stubbed for Phase 2.
12 corpus rows certified vs JVM 1.12.5. make test + shakesmoke green, 0 new
divergences, data.json stays 138/139, selfhost holds.
class / .getClass now return a Class value that renders like the JVM —
(str c)/.toString -> "class <name>", pr -> "<name>", .getName/.getSimpleName/
.getCanonicalName work — but stays = and hash equal to its name string, so
(= (class x) String), class-keyed maps, multimethod dispatch on class, and
instance? keep working against the bare class-name tokens. instance? unwraps
a Class passed as the type arg.
clojure.test/class-match? no longer assumes (class e) is a string (a jolt-ism;
on the JVM a Class isn't a string either) — reads the name via .getName.
Matches JVM Class.toString, which libraries surface in error messages
(clojure.data.json DJSON-54 expects "...of class java.net.URI"). data.json
suite 139/139 bar the one UTF-16 surrogate test (Unicode-scalar char model).
5 corpus rows certified vs JVM; make test + shakesmoke green, 0 new divergences.
java.util.Calendar (a UTC calendar over epoch-ms): getInstance, the int
field constants, set/get/setTime/getTime/getTimeInMillis. java.time pieces
for a zoned round-trip: DateTimeFormatter/ISO_ZONED_DATE_TIME, formatter
.withZone/.parse, LocalDateTime/parse (with formatter), Instant/from,
.toLocalDate/.atStartOfDay. java.sql.Date is now a distinct class (its own
host value + dispatch tags) so a protocol extended to both java.util.Date
and java.sql.Date routes a sql.Date to its own impl. All UTC-consistent.
data.json print-sql-date + print-time-supports-format pass (suite 137/139).
Replace the 33-line pprint shim with a column-aware pretty printer and a
Common Lisp cl-format engine, ported from the ClojureScript implementation
(no STM — atom-backed fields) and adapted to JVM-Clojure interop. Provides
pprint/write/write-out/with-pprint-dispatch/formatter-out/cl-format and
simple/code dispatch.
core print routes column-aware into an active pretty-writer via a __write
hook (suppressed inside with-out-str captures); PrintWriter host class
forwards into the wrapped writer. Re-mint: pprint is baked into the seed.
Unblocks clojure.data.json/pprint (its pretty-printing test passes).
Directives ~R/~P/~C/~F/~E/~G/~$/~(~) not ported (unused by the targets).
make test + shakesmoke green, 0 new divergences, selfhost holds.
Shaking out clojure.data.json's own test suite (now 134/137):
- Reader.read(char[],off,len) bulk read + PushbackReader.unread(char[],off,len)
on the string/pushback reader jhosts; instance? java.io.PushbackReader/
Reader/StringReader (data.json re-wraps a reader unless it's already a
PushbackReader, so this is load-bearing for repeated reads).
- number protocol dispatch by actual type: a flonum is Double (not Long),
exact ratio is Ratio, exact integer is Long — value-host-tags split.
- Integer/toHexString|toOctalString|toBinaryString|toString; .isNaN/.isInfinite
as instance methods on numbers.
- EOFException ctor/class; .isArray on a class-name string.
- dispatch tags for the uuid/bigdec/inst host values so a protocol extended to
java.util.UUID / java.math.BigDecimal / java.util.Date / java.time.Instant
reaches its impl; canonical-host-tag strips java.math./java.time.
- instant/zoned/local time values compare = by epoch-ms (two parsed Instants).
- java.time.Instant/parse, java.sql.Date ctor + valueOf, TimeZone/getDefault,
DateTimeFormatter/ISO_INSTANT.
All runtime .ss (no re-mint). 9 corpus rows certified vs JVM; make test +
shakesmoke green, 0 new divergences.
- String.getChars copies into a char-array; String. builds from a char[]
(whole or offset/count slice); subSequence returns the substring.
- str of a StringBuilder returns its content (was the opaque host object;
.toString already worked).
- Appendable.append gains the 3-arg subsequence form append(csq,start,end)
on StringBuilder/StringWriter/file/port writers.
- reader combines a \uXXXX surrogate pair into the one Unicode scalar
(😃) instead of crashing on the lone high surrogate; a stray surrogate
maps to U+FFFD. (A high-plane char re-escaped as \uXXXXX stays the
irreducible UTF-16/scalar divergence.)
- TimeZone/getDefault.
These let clojure.data.json read and write JSON; its own suite goes from
not loading to 97/133 assertions passing (remainder: per-type writer
dispatch for uuid/bigdec/date, EOFException, niche date interop).
(instance? clojure.lang.Named :a), java.lang.CharSequence/Number,
java.util.Map/Set/List/Collection, clojure.lang.Associative now report
true for jolt's value model, matching the JVM (a Map is not a Collection;
a List excludes sets/maps). Libraries branch on these — data.json's
default-write-key-fn keys on clojure.lang.Named, so map keys serialized
with the leading colon before this.
A slash-free dotted symbol with a Capitalized final segment (java.util.Map,
clojure.lang.Named, java.time.Instant) now self-evaluates to its name string
instead of resolving to nil — jolt models a class as its name, so a library
can extend a protocol to, or instance?-check, a host class jolt has no shim
for. hc-resolve-global classifies these as :class; the analyzer emits a const.
extends? now matches when either the query or the registered tag is a dotted
suffix of the other, so (extends? P java.util.Collection) finds the impl
extend registered under the canonical short tag.
Add DateTimeFormatter/ISO_INSTANT (UTC, trailing Z).
These unblock loading clojure.data.json, which dispatches JSONWriter on
java.util.Map/Collection/CharSequence/Instant and defaults a formatter to
ISO_INSTANT.
read-string/read now return real sets for #{...} literals (top-level and
nested) instead of the reader's {:jolt/type :jolt/set} form — the data
seams convert set forms to sets (recursing, preserving metadata and source
map-key order); clojure.edn already did this. The compiler keeps reading
via the raw reader, so set literals in code stay forms the analyzer lowers.
format %x now emits lowercase hex (Chez number->string is uppercase); %X
unchanged.
extend and extends? handle a nil target type (host tag "nil"), matching
extend-type — protocols can be extended to nil via the function form, not
just the macro.
Found porting transit/data.json and shaking out aero.
conj/assoc/dissoc/disj/pop/into and empty now thread the receiver's
metadata onto the result, matching Clojure (each op constructs a new
collection with meta() carried forward; coll.empty() is
EMPTY.withMeta(meta())). The metadata side-table is now weak so meta on
intermediate collections is reclaimed with them, and empty-list-t carries
an (unused) field so a metadata-bearing () is a distinct identity from the
shared singleton instead of leaking meta onto every ().
Unblocks metadata-driven walks (aero/integrant): (into (empty form) ...)
now preserves a vector/map/set's metadata, so a postwalk whose outer fn
reads (meta x) sees it.
The reader lowered ^meta on a vector/map/set literal to a runtime
(with-meta form meta) list, so read-string/edn of data with metadata
returned the form and lost the metadata. Attach it to the value instead,
as Clojure does; the analyzer re-emits (with-meta coll meta) for a
meta-carrying collection literal in code, so a literal still carries its
metadata at runtime and ^Type/^long arglist hints (consumed by
analyze-arity directly) are unaffected.
Also: pr honors *print-meta*, and clojure.walk/clojure.edn re-attach
metadata to the collections they rebuild (matches Clojure; a
metadata-driven config lib like aero relies on it).
clojure.test reported a host-condition crash with a blank message (ex-message is
nil for non-ex-info conditions); fall back to jolt.host/condition-message.
condition-message itself left a Chez format-template message (open-input-file's
'failed for ~a: ~(~a~)') unformatted; apply its irritants.
jfile lacked .isAbsolute (path starts with /). clojure.core/default-data-readers
and *data-readers* were nil, so a library merging them (aero's reader opts)
couldn't resolve #inst. Define them keyed by symbol, like Clojure.
(instance? clojure.lang.IObj/IMapEntry/IRecord/IPersistentMap/... x) all returned
false, so libraries branching on these interfaces (e.g. a metadata-preserving
walk guarded by IObj) silently misbehaved. Register an instance-check arm mapping
the interface tokens to jolt predicates, matched by last dotted segment.
(tagged-literal t f) compared by identity, so two equal tagged literals were not
= and didn't dedup as map keys / set members; JVM's TaggedLiteral is value-equal.
Add a register-eq-arm! for jtagged (tag + recursive form); hash is already
structural. Fixes an infinite loop in any fixpoint that dedups structures
containing tagged literals (e.g. aero's #ref expansion).
io/file already resolved a relative path against JOLT_PWD (the user's cwd before
the launcher cd's to the repo root), but slurp and io/reader on a relative string
path opened it against the process cwd — so a program run from its own directory
couldn't read its own relative files (e.g. aero's (read-config "config.edn")).
Route the string branches through project-relative, matching io/file. Internal
callers pass already-resolved paths through read-file-string and are unaffected.
clojure.core/*print-meta* was missing, so (binding [*print-meta* true] …) failed
to compile ("var of non-var"). Add it (default false, like the JVM); corpus
covers binding it.
The EBNF and reader S7 already specified ::kw auto-resolution — the
implementation was out of spec, now aligned. S7 noted unknown ::alias/k MUST be a
read error; jolt is lenient (reads :alias/k), so record that as a deviation.
Corpus gains JVM-certified rows for ::kw resolution, clojure.edn :default
receiving a symbol tag, and with-meta on a lazy seq.
A pushback-reader (drainable, so clojure.edn/read over it works); getLineNumber
is a stub since jolt doesn't track line numbers. Config readers (aero) wrap their
input in one.
A (defmethod m …) where m is :refer'd from another ns passed the bare symbol to
defmethod-setup, which resolved it in the current ns and created a shadow multifn
— the method never reached the real one. Resolve an unqualified name through the
refer table (then current ns) so it lands on the referred multifn.
The AOT build strips the ns form, so the refer table is empty in a binary; emit
chez-register-refer!/-refer-all! per app ns alongside the existing alias
registrations. build-app's fixture gains a defmethod on a referred multifn.
with-meta/meta only recognized the eager collections, so (with-meta (map-indexed
…) m) — a lazy seq — threw "value does not support metadata". A lazy seq is IObj
on the JVM; add it to the metadatable set (keyed in the side-table by identity,
like cseq/procedure).
The reader dropped the namespace on ::kw (read ::foo as :foo), so auto-resolved
keywords never matched their qualified form — code that round-trips them (spec
keys, aero's :aero.core/* expansion keys) silently broke. Resolve ::name against
the current ns and ::alias/name through the alias table, as Clojure does. The
runtime loader reads form-by-form with the ns set after the ns form; the
cross-compile reads all forms up front, so ei-emit-ns*/ei-emit-ns-records set the
ns before reading.
clojure.edn/read over a reader discarded its opts map — :readers/:default/:eof
were ignored, so a custom :default never saw the tag. Route the reader arity
through read-string so opts apply, and pass the tag to :default as a symbol (not
the internal :#name keyword), matching Clojure.
Seed re-minted (the ::halt transducer key in clojure.core now reads as
:clojure.core/halt). Corpus gains ::-keyword rows; the unit case that asserted the
old ns-dropping behavior now asserts the qualified result.
types.clj held the inferencer, the success-type checker, and the driver in one
716-line namespace. Move the self-contained checker into jolt.passes.types.check:
the error-domain predicates (not-number?/not-seqable?/not-callable?), the op
tables, type-name, check-invoke, and the user-fn registry. These are pure over
inferred types and the run's env cells, with no inference, so a check-rule edit
can no longer perturb the inferencer.
The infer-coupled probes stay in types.clj — isolated-diag-count and
check-user-call re-run inference, so moving them would make check depend on the
inferencer and reintroduce the cycle. Verbatim move; new ns wired into
ei-compiler-ns-files; seed re-minted to the byte-fixpoint.
The stderr seam wrote to current-error-port without flushing, so a process that
never returns from -main (a server loop) left its log lines in a buffer that only
drained at exit — they never appeared. Flush each write, like the JVM's
auto-flushing System.err.
The source loader sets the current ns and registers :as aliases per file. The
build flattened every app namespace into one image with no such markers, so all
app forms ran under the last-set ns ("user"). Two breakages followed, both only
in a built binary:
- defmulti/defmethod resolve their target var through chez-current-ns, so they
registered the multifn under "user" while compiled var-derefs used the baked
ns — the multifn the app saw was uninitialized ("not a fn nil" on dispatch).
- a quoted alias-qualified symbol (a (defmethod ig/foo …) on an aliased multifn)
resolves its ns through chez-resolve-alias, but the stripped (ns …) form left
the alias table empty, so it landed in ns "ig".
bld-ns-prelude now emits (set-chez-ns! ns) plus chez-register-alias! for each
ns's :as aliases before that ns's forms, in both the normal and tree-shake emit
paths. The build-app fixture gains a :default multimethod and an aliased cross-ns
defmethod so buildsmoke covers both across all build modes.
walk treated a record as a plain map (record? implies map?), rebuilding it via
(into (empty form) ...) which yields a bare map and drops the type. Add a record
branch before the map branch that conj-es the walked entries back onto the
original, matching JVM clojure.walk's IRecord case. Type-dispatched walks need it
— integrant resolves #ig/ref by detecting its Ref record while postwalking the
config, so without this every ref silently fails to resolve.
clojure.walk is baked into the prelude, so the seed is re-minted. Corpus gains
five JVM-certified rows for record type/instance? survival through pre/postwalk.
infer's :invoke case was ~120 lines of cond arms hand-coding eight call
patterns, all destructured positionally with (nth r 0)/(nth r 1) on the
[type node'] tuples infer returns. Split each pattern into a named helper
(infer-pred-fold/-kw-lookup/-get-lookup/-reduce-hof/-seq-hof/-conj-into/-call)
behind an infer-invoke dispatcher that keeps the cond guards verbatim, and add
ty/nd accessors for the tuple so a silent transposition can't hide.
The accessors are applied only to genuine infer results (the new helpers and
infer-fn-seeded); the :map/:let/:loop arms interleave non-infer pairs
(binding tuples, accumulator pairs) with infer results, so those keep nth.
Pure restructuring — the guards, order, and bodies are unchanged; seed
re-minted to the byte-fixpoint, gate green, 0 new corpus divergences.
The inliner and the type inferencer each recognized (:k m) and (get m :k)
lookups with their own copy of the callee tests — the get-callee check was
duplicated verbatim across both. Lift kw-callee?/get-callee? into
jolt.passes.fold (alongside scalar-const?) and call them from both passes so
the head recognition can't drift.
Only the head predicates move. The deliberate differences stay: the inliner
still accepts any scalar key in the get-form (its scalar-replacement targets
can be string/number-keyed maps) while the inferencer requires keyword keys
for struct field typing, and the inferencer keeps its two arms separate so each
rebuilds args for its form. The backend's value-as-fn ifn-kind is left alone.
numeric.clj dbl-spec/lng-spec and backend_scheme.clj dbl-ops/lng-ops must agree —
a spec'd op with no table entry makes emit-numeric splice a nil op string. Document
the contract on both sides. Comment-only; seed re-mints byte-identical, gate green.
The other two ideas in this bead were rejected after inspecting the code:
- collapsing inline.clj local-escapes? onto reduce-ir-children would reintroduce the
under-reporting hazard its docstring deliberately guards against (default-true is
load-bearing for scalar-replacement soundness).
- folding numeric recur-kinds/recur-arg-lists into one walk loses the type-env
threading recur-kinds needs through :let; the parallel split is justified.
analyzer.clj referred jolt.host/form-char? but never called it (form-char? stays
live — backend_scheme.clj uses it). Promote numeric.clj an-invoke's :wild operand
rule (an integer literal is valid in either fl/fx kind) from an inline comment to the
function docstring. Both output-neutral: the seed re-mints byte-identical, gate green.
Two small clarity moves from the review:
- dynamic-vars.ss -> dynamic-var-defaults.ss. It holds the default VALUES of a few
core dynamic vars (*clojure-version*, *assert*, …); the near-identical dyn-binding.ss
holds the binding-stack machinery. The names were easy to confuse.
- Pull the ~60-line %-format engine out of the natives-misc.ss grab-bag into
natives-format.ss (its ->long/pad-left/fmt-float helpers were local to it).
rt.ss loads + MODULES.md updated. Runtime .ss, no re-mint; make test green, format +
the dynamic vars verified.
natives-parity.ss was a grab-bag (its own header called it 'parity'). Dissolve it
into homes that say what they hold:
- cat -> natives-transduce.ss (was natives-xform.ss, renamed for the
whole transducer surface: volatiles + cat + transduce/sequence)
- transient? -> transients.ss
- rseq -> natives-seq.ss
- hash family -> natives-misc.ss (the public hash API over jolt-hash)
The remainder — the #?() feature set, the reader-conditional + re-matcher tagged-map
ctors, and macroexpand — is a coherent reader/macro runtime-support unit, kept as
natives-reader.ss (np- prefix -> nr-). rt.ss loads + two comment refs updated.
Runtime .ss, no re-mint. make test green; hash/transient?/rseq/cat/macroexpand and
#?() reader features all verified.
The trio split on a fine axis (registry core / statics / object classes) but the
names didn't say so — 'static-statics'/'static-objects' and headers that read
'Continues X'. Rename:
host-static-statics.ss -> host-static-methods.ss (Class/member statics + fields)
host-static-objects.ss -> host-static-classes.ss (instantiable object classes)
host-static.ss stays the registry core. Headers rewritten to state each file's role
and what it covers instead of chaining. rt.ss loads + the one comment reference +
MODULES.md updated. No code moved; runtime .ss, make test green.
Under --direct-link a top-level def binds jv$<fqn> and app->app calls bound directly
to it. emit-invoke raw-applied that binding for any var callee, but only a fn-valued
def is a Scheme procedure: (def cfg {...}) then (cfg :a) emitted (jv$cfg :a), applying
a pmap -> "attempt to apply non-procedure". Maps/sets/keywords are invokable in Clojure
via jolt-invoke, which the indirect path used, so this only bit closed-world builds.
Track which direct-linked vars hold fn literals (direct-link-fns, registered at the def
site when the init op is :fn) and only raw-apply those. A non-fn callee falls through to
the jolt-invoke branch, which still uses the direct jv$ binding as the invoke target —
so the var-deref is still skipped, just not the dispatch.
Seed source: re-minted. Regression in directlink-test.ss (jolt-cw1o).
The printer's two entry points (jolt-pr-str in rt.ss, jolt-pr-readable in printing.ss)
get register-pr-str-arm! / register-pr-readable-arm!, plus register-pr-arm! for the
types whose str and readable forms match (bigdec/inst/uuid/tagged/record/ns/var). The
normalize arms (sorted, lazy-seq, queue) and the uri readable arm register per-printer.
Also folds in the hash (dyn-binding var-cell), class (io uri/uuid/file), and get
(transients) arms missed earlier.
natives-array's get stays a case-lambda wrapper on purpose: its 2-arg path errors on
an out-of-bounds index while the 3-arg path returns the default, an arity distinction
the (coll k d) registry collapses — left as-is to preserve behaviour.
Completes jolt-lmot: all six dispatchers (hash/class/get/=/pr-str/pr-readable) off the
set!-rebind chains. make test green, 0 new corpus divergences; pr-str/str of inst,
uuid, bigdec, sorted-map, record-with-lazyseq, queue all verified.
jolt-get (4 sites: host-table/natives-misc/inst-time/records) and jolt=2 (6 sites:
records/vars/inst-time/natives-misc/bigdec/host-table) move off the set!-rebind
chains to register-get-arm! / register-eq-arm!. get's case-lambda becomes a stable
2/3-arg entry over a 3-arg dispatch; the equality arm pred is (a b) since either arg
may carry the type, and the host-table sorted arm normalizes-then-re-dispatches.
Behaviour-preserving (runtime .ss): var/inst/bigdec/record/uuid equality, record!=map,
sorted-map=plain-map, and all the get cases verified; make test green, 0 new corpus
divergences. Four of six dispatchers done; the printer (pr-str/pr-readable) remains.
Replace the set!-capture-and-rebind chains extending jolt-hash (3 sites: inst-time/
host-table/records) and jolt-class (3 sites: bigdec/natives-queue/host-table) with a
register-hash-arm! / register-class-arm! registry (the pattern already used by
register-str-render!). The base dispatcher walks its arms — disjoint types, so order
is immaterial — then falls to the base cases. The entry is stable, so the per-site
(def-var! "clojure.core" "class" …) re-points are gone.
Behaviour-preserving (runtime .ss, no re-mint): jinst hash, bigdec/queue/record
class, record hash, and a sorted-map hashing as its plain map all verified; make test
green, 0 new corpus divergences. First two of six dispatchers; get/=/pr-str follow.
set-optimize!/set-direct-link! are process-global flags in the back end, set then
reset around the emit. A strict-form build error (failing forms error the build by
design) skipped the reset, leaving the compiler in optimize/direct-link mode for any
later in-process caller. dynamic-wind guarantees the revert. Behaviour unchanged on
the success path; both --tree-shake and --opt --direct-link build + run identically.
Tree-shaking was split across emit-image.ss (the dce-* helpers + record producer)
and build.ss (bld-shake-all + the manifest splice), with the DCE record accessed by
raw (vector-ref r 0..3) in ~10 places and the manifest splice/drop driven by
substring-matching (load "…prelude.ss").
- New host/chez/dce.ss owns the whole DCE concern: a named record API
(dce-rec/-keep?/-fqn/-refs/-str — no more positional vector indexing), the ref
extraction + ref-set constants, dce-blob-records, and dce-shake decomposed into
dce-build-graph / dce-reachable / dce-bail-scan / dce-partition (was one 50-line
bld-shake-all doing five jobs with shared mutable state).
- emit-image.ss keeps only ei-emit-ns-records (it drives the ei-* compiler) and uses
the dce-rec constructor.
- The runtime manifest is now tagged ('prelude/'image/'compile-eval); bld-emit-runtime
dispatches on the tag instead of substring-matching file paths, so the core-splice
and compiler-drop can't silently break on a rename/reorder.
Behaviour-preserving (runtime .ss, no re-mint): build-app shakes identically
(56/460, 8.12MB), make test green, make shakesmoke green (4 git-lib apps
byte-identical, same sizes).
Navigability groundwork from the architecture review — zero behaviour change.
- docs/MODULES.md: the repo map. Area -> directory -> key files -> re-mint?, plus
per-feature touch points (tree-shaking, direct-linking, numeric fl/fx, inlining,
multimethods, deps) and where a given clojure.core fn lives. Answers "where does
X live / what's related to Y" in one read.
- docs/rfc/README.md: index the 7 RFCs; flags RFC 0007's stale "no code yet" status
(direct-linking + tree-shaking shipped) and the undocumented inlining/numeric work.
- CLAUDE.md: document the var-deref calling convention (public defns reached from the
.ss runtime by string lookup aren't dead), the def-var! native pattern, and the
overlay shadowing rule; point at MODULES.md.
- REFACTOR_PLAN.md: the prioritized, risk-tiered plan (working doc for this branch).
README + tools-deps.md cover --tree-shake and --direct-link: what tree-shaking does
(whole-program reachability over app + libraries + clojure.core, drop unreachable,
drop the compiler for no-eval apps), and why it bails to keep-all when reachable code
resolves vars by name at runtime (eval/resolve/ns-resolve/...), with the diagnostic
output and how to make an app shakeable. Notes the Stalin soundness model.
When --tree-shake keeps everything because reachable code resolves vars at runtime,
list the offending def -> bail-ref pairs (up to 6) instead of just saying it
skipped. Makes it actionable: e.g. ring-app shows
clojure.tools.logging/call-str -> ns-resolve and selmer.filters/generate-json ->
resolve, so you know which library (not your code) blocks shaking.
--tree-shake now shakes the clojure.core/stdlib prelude in the same reachability
graph as the app + libraries — only core fns actually reached from -main ship.
dce-blob-records reads prelude.ss with Chez `read`, unwraps each
(guard ... (def-var! "ns" "name" V)) and builds the core->core call graph from the
(var-deref/jolt-var "ns" "name") refs in V — the real emitted edges, no
re-analysis. bld-shake-all merges core + app records into one BFS; roots are -main,
side-effecting forms, and the clojure.core fns the runtime .ss shims reference by
name (enumerated in dce-runtime-core-roots). The shaken core is spliced where the
prelude.ss blob was, in original (tier) order, so load-time deps are preserved.
Bail (reachable runtime resolve) keeps the prelude whole.
Soundness follows Stalin's rule (any reference, value or call, keeps a def live):
dce-collect-refs counts :var and :the-var; non-def registration forms are always
kept (covers protocol/multimethod dispatch). Validated by make shakesmoke: the four
deps.edn git-lib apps build byte-identical output shaken vs not.
Wins (binary): build-app 9.84MB -> 8.12MB (dropped 403/457 core defs); malli-app
10.0MB -> 8.1MB; markdown 9.9 -> 8.3; commonmark 9.8 -> 8.1 — all output-identical.
build-smoke asserts an unused core fn (group-by) is dropped; full make test green.
Two things, both from studying Stalin's closed-world DCE:
1. Soundness fix: dce-collect-refs now counts a :the-var (#'x / (var x)) reference,
not just a :var. Stalin's rule is that ANY reference — value position, not only a
direct call — keeps the target live; a var referenced as #'x would otherwise be
wrongly dropped. (My :var collection already covered value-position refs; this
closes the the-var hole.)
2. host/chez/tree-shake-smoke.sh (make shakesmoke): builds example apps default vs
--tree-shake and requires identical output — the real risk is shaking a binary
that pulled libraries via deps.edn. Covers markdown/malli/commonmark/hiccup
(git-lib apps). All produce byte-identical output shaken vs not, and drop
~0.8-1MB (malli 10.0MB -> 9.0MB) from the compiler-drop. Slow; not in the default
gate. Skips without the examples repo.
An AOT-compiled app only needs the analyzer/back end at runtime to compile from
source — eval / load-string / load-file. Macros are expanded at build time and a
require of a baked namespace no-ops, so a closed app that never compiles at runtime
carries the compiler image (~0.8MB) as dead weight.
Under --tree-shake, when reachability is trustworthy (no bail) and no reachable code
references eval/load-string/load-file/load-reader/load, omit host/chez/seed/image.ss
+ compile-eval.ss from the runtime manifest. bld-tree-shake returns the flag
alongside the shaken forms; bld-emit-runtime filters the manifest.
Measured: build-app 9.84MB -> 9.05MB, still runs. Safety verified: an app that evals
keeps the compiler (eval is a bail + compile ref) and eval works at runtime.
build-smoke asserts the compiler is gone in the no-eval app; full make test green.
`jolt build --tree-shake` (or deps.edn :jolt/build {:tree-shake true}) does
reachability DCE over the re-emitted app + library namespaces: keep -main, every
side-effecting (non-def) top-level form, and every def reachable from those; drop
the rest. A macro (expanded at AOT, never called at runtime) is prunable too.
Sound: bails (keeps everything) if REACHABLE code resolves vars by name at runtime
(eval/resolve/ns-resolve/requiring-resolve/find-var/intern/load-string/...), which a
static call graph can't follow. Unreached eval-using library code is simply shaken
away and never triggers the bail. clojure.core and the compiler image stay baked
(prelude + image blobs), so only re-emitted namespaces are shaken for now.
The reachability machinery is in emit-image.ss (records: keep?/fqn/refs/str via
reduce-ir-children) + build.ss (BFS + bail check). build-smoke covers it (drops the
unreachable `twice` macro, output unchanged). Opt-in; default builds are untouched.
full make test green.
Scope note: this shakes the re-emitted app/lib code only. Measurement shows jolt's
compiled code is ~5.8MB of a ~9.8MB binary, dominated by the clojure.core prelude
(~1.5-2MB) and the compiler image (~0.8MB) — both baked blobs this pass doesn't
touch. Those (shake-core, drop-compiler-when-no-eval) are the larger footprint wins,
filed as follow-ups.
A loop var with an integer-literal init now types :long (fx ops) when every recur
arg in its slot is an increment-style step — the var unchanged, inc/dec, or (+/-
var <int-literal>). So (loop [i 0] (recur (inc i))) gets fx1+/fx<? without a hint,
matching how Clojure treats a primitive-long loop counter.
Soundness: only increment steps qualify. A multiplicative or large-growth
accumulator like (recur (* acc 2)) is never seeded, so it stays generic and keeps
arbitrary precision — a bignum-producing loop (e.g. a factorial) is unaffected.
counter-step? gates this; the existing fixpoint demotes anything inconsistent.
test/chez/numeric-test.ss 44/44 (incl. a factorial loop staying bignum-exact while
its counter is fx); full make test green, 0 new corpus divergences.
jolt.passes.inline was fully written but dormant — it fetched bodies via the
inline-ir host hook, which was a stub returning nil. Wire it up: run-passes stashes
each inline-eligible defn (single fixed arity) as its form is optimized, and
inline-ir hands the body back at call sites under --opt.
The catch was the ^double/^long coercion: an inlined fn drops its param-entry and
return coercion, so (work 3 4) on a ^double fn would return 25 instead of 25.0. New
:coerce IR node carries the coercion inside the spliced body — the inline pass wraps
a hinted param's arg and the return in :coerce, the back end lowers it
(exact->inexact / jolt->fx), and jolt.passes.numeric reads its :kind. So an inlined
call matches the called one and the body's fl*/fx* fast path still fires.
Only under --opt (closed world); the seed mint and -e don't inline, so selfhost and
the corpus are unaffected. test/chez/inline-test.ss 12/12 (make inline); full make
test green, 0 new corpus divergences.
Bench (hot loop, body is a ^double helper call): direct-link 500ms -> --opt
(inlined) 184ms = 2.7x, by eliminating the call + coercion wrappers and letting Chez
fuse the fl-ops unboxed. ~26x over the default dispatched build.
A ^double/^long return hint on a fn's name now (a) coerces the fn's value on the
way out — exact->inexact / jolt->fx, like a JVM primitive return — and (b) types a
call to it, so an accumulator over the result specializes:
(defn ^double work [^double x ^double y] (+ (* x x) (* y y)))
(loop [acc 0.0] (recur (+ acc (work a b)))) ; (+ acc (work ..)) -> fl+
The analyzer pushes the name's numeric tag onto each arity (:ret-nhint) for the
back-end coercion, and resolve-global surfaces the callee's declared return
(:num-ret, read from var meta) onto the :var node so jolt.passes.numeric types the
call. defn carries the name hint through.
This unblocks the accumulator-over-fn-result pattern that round 2 had to demote.
The win is bounded by call overhead in an open/dispatched build (~1.15x on a hot
loop whose body is a helper call); it compounds with direct-linking and, later,
inlining. A numeric return hint is a contract, like ^long — redefining the var to
return another type in an open build breaks it.
Not yet: per-arity arglist return hints, (defn f (^double [x] ..)). Gate:
test/chez/numeric-test.ss 39/39; full make test green, 0 new corpus divergences.
loop-kinds only typed :double accumulators; a ^long-seeded loop var (e.g.
(loop [acc start] ...) with a ^long start) stayed generic even though it's sound
to fx-type — :long only ever comes from an explicit hint, and a ^long value is
already coerced to a fixnum at fn entry. Keep the init's kind (:double or :long)
through the fixpoint, demoting only on a recur-arg mismatch.
Integer-literal-init loop vars (a bare (loop [i 0] ...)) still stay generic by
design: :long is never seeded from a literal, so a bignum-producing loop keeps
arbitrary precision.
A loop binding whose init is double and whose every recur arg stays double (a
bounded monotone fixpoint) is typed :double, so its arithmetic — and the recur
args feeding it — emit fl-ops. Chez can then keep the accumulator unboxed in a
float register across the loop.
Integer loop vars stay untyped: a bare integer init never seeds :long (same rule
as round 1), so a bignum-producing loop keeps arbitrary precision rather than
overflowing a fixnum. recur-kinds walks only tail position (if/do-ret/let-body),
stopping at nested loop/fn so a loop sees only its own recur.
A/B on a loop-carried double accumulator: 735ms generic -> 500ms typed (1.47x),
closing the gap to the JVM from ~3.3x to ~2.2x. The integer counter stays generic,
which is most of the residual.
A ^double/^long param hint (or a float literal) now drives Chez flonum/fixnum
ops instead of generic arithmetic — JVM-style primitive hints, available in every
build and at -e (not gated on direct-linking or whole-program inference).
New pass jolt.passes.numeric: a local forward type-flow seeded from ^double/^long
fn-param hints (analyzer attaches :nhints per arity) and float literals,
propagated through let inits / arithmetic / if / do. It tags an arithmetic invoke
:num-kind :double|:long when every operand is that kind (an integer literal is a
wildcard, coerced to a flonum in a double op). The back end lowers a tagged node
to fl+/fl-/fl*/fl//fl<?/... or fx+/fx*/fx1+/fxquotient/... (unchecked-add etc.
join the fixnum path; == too). Runs last in run-passes, both branches.
Soundness: :long is seeded ONLY from an explicit ^long hint, never a bare integer
literal, so un-hinted integer code keeps jolt's arbitrary-precision numbers — no
fixnum-overflow surprise, no corpus divergence. :double comes from ^double hints
and float literals (flonum arithmetic is always flonum, matching the generic
result). A ^long hint is a promise the value is a fixnum: fx+ raises on overflow,
like a JVM fixed-width long.
Numeric-hinted params coerce at fn entry (exact->inexact / jolt->fx), the way the
JVM coerces a primitive parameter — so the body's fl*/fx* ops can rely on the
type even when a caller passes an exact int (e.g. Chez's (* 0 1.0) => exact 0).
Round 1 specializes hinted straight-line / fn-body arithmetic. fl-ops are ~4x
generic in a tight Chez loop, but realizing that on loop-carried accumulators
needs loop-var typing — round 2. Sound foundation, gated by test/chez/numeric-test.ss.
Release builds can legitimately want runtime dynamism (redefinition, eval,
load-string), so closed-world direct-linking shouldn't be forced on them. Gate it
behind an explicit --direct-link flag (or deps.edn :jolt/build {:direct-link
true}); off by default in every mode, including release and --opt.
build-binary takes an explicit direct-link? arg instead of deriving it from the
mode. build-smoke now covers the --direct-link path and asserts the cross-ns call
actually lowers to a jv$ binding; default release stays dynamically linked.
A release/optimized `jolt build` is a closed world: every app def is final, so
an app->app call can bind to the def's Scheme binding directly instead of going
through (jolt-invoke (var-deref ns name)).
The emitter gains a direct-link mode (off for the seed mint, runtime -e/repl, and
dev builds). With it on, a top-level app def also emits a binding jv$<ns>$<name>
that def-var! aliases; an app->app call or value-ref to a name already emitted in
the unit lowers to that binding, skipping both the var-table lookup and the
generic IFn dispatch. ^:dynamic/^:redef defs and nested defs (a defonce's inner
def) opt out and stay indirect. Off direct-link mode, emit-top-form is exactly
emit, so the seed and runtime eval are byte-unchanged (selfhost holds).
build.ss turns it on for release + optimized; the defined-set accumulates across
the dependency-ordered namespaces so a dep's defs are linkable by the time the
entry that calls them is emitted. App->core calls stay indirect for now (core is
the baked seed); that's a later stage.
~1.74x on a hot cross-namespace call loop (26.5s -> 15.2s).
Build output landed in the CLI's cwd (the jolt repo, since bin/joltc cd's
there), not the project — so a bare -o path or the default binary appeared
in the wrong place. Resolve output against JOLT_PWD, and default it cargo-
style under the project's target/: target/release for release/--opt,
target/debug for --dev, named after the project dir. The <name>.build scratch
dir sits beside the binary, so it lands under the same target dir. -o is
honored — absolute as-is, relative against the project.
A built binary dropped its deps.edn :jolt/native declarations and its
resource roots, so an FFI+resources app (ring-app) failed at runtime:
sockets/sqlite gave 'no entry for socket' and io/resource returned nil.
The buildsmoke fixture is pure compute, so neither path was exercised.
The launcher now loads required + :process native libs before the app's
top-level forms (a library's defcfn resolves its foreign-procedure symbols
at top-level eval during startup, so the libs must be loaded first);
optional libs load in the scheme-start launcher, where a missing lib is
caught rather than aborting the heap build.
deps.edn :jolt/build {:embed [dirs]} bakes those dirs' files into the heap
(register-embedded-resource! at heap build), so io/resource serves them with
no files on disk. Non-embedded resources resolve at runtime against JOLT_PWD,
and io/file reads (e.g. config.edn) stay external.
build-binary now takes the encoded natives, embed dirs, and project paths
from cmd-build; deps/resolve-project surfaces them. Buildsmoke fixture grows
an embedded resource + a :process native to cover both paths.
jolt could call C (foreign-fn -> foreign-procedure) but C could not call back
into jolt, which GTK signals (and any callback-taking C API) require. Add the
inverse: jolt.ffi/foreign-callable wraps a jolt fn as a C-callable function
pointer, mirroring the foreign-fn pipeline.
A new jolt.ffi/__ccallable special form carries the fn as a child expression
(analyzed + walked by the passes; ir.clj gains an :ffi-callable arm in both
child walks) plus literal arg/ret type keywords. The back end lowers it to a
locked Chez foreign-callable and returns its entry-point address as a jolt
pointer; host/chez/ffi.ss registers the code object so the collector keeps it,
and free-callable unlocks it. :collect-safe emits the convention that
reactivates the thread on entry, for callbacks fired while it is parked in a
:blocking call (a GTK main loop).
Test: ffi-binding-test.ss sorts an int array through libc qsort with a jolt
comparator (C -> jolt -> C). Re-minted seed.
Dead code removal, O(n^2) hot-path fixes (transients/queues/deps), wiring the
optimization pass pipeline into compile + build, deterministic seed emission,
splitting the oversized files (20-coll, host-static, records, types lattice),
and restructuring the two worst maintainability smells: the str-render /
instance-check set!-override chains became registries, and the type-inference
walk now threads an immutable env instead of ~14 module atoms.
Adds an inference gate (make infer, 26 cases) so the type pass — which the
corpus/unit gates don't exercise — has real coverage. Full gate green:
self-host fixpoint holds, corpus 2735 (0 new divergences).
Adds 3 cases for the opt-path checker (set-check-mode! -> run-inference ->
take-diags!): diagnostics drain once, the buffer empties on re-drain, and
check-mode off produces none. This is the public checker entry the other 23
cases didn't touch. Runtime-only.
types.clj drove inference through ~14 module-level atoms; the infer walk was
non-reentrant and depended on hidden set-*! install order. Thread one immutable
env (mk-env) through infer instead: it snapshots the installed config
(rtenv/vtypes/record-shapes/protocol-methods/map-shapes?) and carries the
per-run flags and accumulator/guard cells (diags/calls/checking-set/diag-memo).
A fresh env per run makes the pass re-entrant — isolated-diag-count's probe now
runs under a sub-env with its own diags cell instead of save/restoring a shared
atom.
Only state whose lifecycle spans separate API calls stays module-level: a
config-box the set-*! API writes, the escapes/user-sig sweep registries, and a
bridge holding the last checking run's diags for take-diags!. record-type-from-
entry/field-type-from-tag now take the shapes map directly rather than reading a
global.
jolt-ogib.10. Behavior pinned by the new infer gate (23 cases) plus selfhost +
buildsmoke. Re-minted seed.
The corpus/unit gates compile through run-passes' const-fold-only branch, so
the type-inference walk runs only under jolt build --opt — buildsmoke hit one
trivial app and checked stdout. run-infer.ss drives the pass directly: analyze
a source string, then call check-form / infer-body / the set-*! registries and
assert diagnostic counts and collected calls/escapes. Wired into make ci.
Gives the inference pass real behavior coverage so refactoring its internal
state is gate-validatable. jolt-ogib.10 groundwork.
jolt-str-render-one and instance-check were each extended by a chain of
set!-wrapping closures spread across ~10 and ~5 host files, so the real
behavior of either was scattered and load-order-dependent. Give each a
registry the base file owns: converters.ss/records-interop.ss define the
registry plus a register-* helper, and each extending file registers one arm
instead of capturing %prev and set!-ing the global.
str-render arms are type-disjoint; instance-check arms run newest-first (the
old outermost-wins order) and may return 'pass to defer. The string-token ->
symbol normalization the natives-array arm did for every inner arm moves to
the dispatcher head; array tokens stay strings for that arm to decide.
jolt-ogib.14. Runtime-only shims, no re-mint.
The lattice-split commit staged its seed before make remint ran, so image.ss
lagged the source. Commit the correct re-minted image (gensym renumbering only).
types.clj was 852 lines mixing the pure structural-type algebra with the
inference engine, checker, and driver. Move the lattice — scalar/struct/vec/set/
union types, join-t, depth-cap, shape, and the numeric/vector return-fn sets —
into jolt.passes.types.lattice (no inference state, no requires). types.clj
requires it; the engine is now ~720 lines. Compiled into the image before
jolt.passes.types. Re-minted seed differs only by gensym label renumbering.
records.ss mixed the record/protocol/reify model with JVM exception emulation.
Move the contiguous ex-info accessors, the exception supertype hierarchy,
instance-check, case-string, and the instance-check def-var into records-interop.ss,
loaded right after records.ss. Those are only called at runtime, so the relocated
forms resolve fine; records.ss is now the value model and protocol dispatch.
Runtime shim — no seed change.
The 929-line interop registry split three ways: host-static.ss keeps the
registries, the jhost record, the emit entry points and coercion helpers;
host-static-statics.ss holds the java.lang/util static-method registrations;
host-static-objects.ss holds the host object classes (ArrayList, HashMap, the
reader/writer/tokenizer shims, ctors, URL codecs) and the instance? hook. rt.ss
loads the three in order. Runtime shim — no seed change.
The 1123-line collection tier is the largest source file. Cut it at two existing
section banners into 20-coll (predicates, printing, hierarchies, pure-over-core
leaves), 21-coll (rand/sort seams, the test runner, fn combinators), and 22-coll
(canonical Clojure ports, transduce/into, JVM-shape stubs). No macros in this tier,
so order is the only constraint; the emit-image manifest lists the three in
sequence. Re-minted seed is identical apart from gensym label renumbering.
check-user-call rebuilt the all-:any env once per parameter (O(params^2)) and
re-inferred a callee body at every call site. Build the env once and memoize each
probe by [key i argtype] (and the baseline by [:base key]), cleared per form in
check-form. The global type-env is stable within a form's check and the probe's
calls/escapes side effects aren't read there, so a skipped repeat is observably
identical. (The inline-side re-walk the audit flagged is moot: hc-inline-ir is a
no-op on Chez, so try-inline never reaches body-size/body-closed?.)
ei-emit-ns (emit-image) and bld-emit-ns (build) were near-verbatim copies that had
drifted: the minter guard-wraps and skips failing forms, the build is strict, and
since the passes were wired the build also runs run-passes. Fold both into
ei-emit-ns* with optimize?/guard? flags; ei-emit-ns and bld-emit-ns become one-line
callers. Output is byte-identical (selfhost fixpoint and build smoke stay green).
- str-join delegates to str-join-strs (it only adds the per-element render).
- loader: extract resolve-on-roots; find-ns-file and load both use it.
- NumberFormat registers its short + FQ names from one shared member list.
- inst-time's private floor-div/floor-mod renamed inst-floor-* so they don't read
as the math.ss reals version.
Left the fold/inline/types pure-fn sets and the keyword/symbol ns builders alone:
those are file-local and semantically distinct (e.g. the intern key uses NUL, not
"/"), so merging them would be wrong.
- take-last / drop-last return seqs, not vectors: take-last wraps in seq; drop-last
is the JVM (map (fn [x _] x) coll (drop n coll)) form (lazy, () when empty).
- cycle is lazy ((lazy-seq (concat coll (cycle coll)))) so it no longer counts its
argument and terminates on a lazy/infinite input.
- fold's foldable-call catch uses :default, matching the rest of jolt-core and
also catching a raw host condition from a folding primitive.
- alts! rejects non-channel ports with a clear error (put specs / :default are
unsupported) instead of crashing inside ac-poll!.
- Misc: drop the unreachable second getCause clause; jolt-nth on a string raises
'nth "index out of bounds" like the vector branch; name the inline fixpoint cap;
bld-sh-capture rejoins lines with newlines; clarify a couple of comments.
map-ir-children single-sourced the child layout for rewrite passes; the read-only
analyses each re-enumerated ops by hand. Add a fold companion, reduce-ir-children,
and rebuild body-size, pure?, and body-closed? on it (each reduces to a leaf value
+ the special ops it actually needs). local-escapes? stays an explicit walk — its
default is conservatively true and it inspects node shape beyond child purity, so
folding an unhandled op over its children would be unsound for scalar replacement.
analyze-special inlined def (~35 lines) and set! while try/letfn/fn* were already
helpers. Pull both out and move field-head? above analyze-special so its set! arm
and analyze-list reach it without a forward reference — the file's "only analyze
is forward-declared" invariant holds again. Pure code motion.
The fold/inline/types passes and the jolt.passes façade were baked into neither
seed half and never invoked: compile-eval and build went analyze -> emit directly,
and `jolt build --opt` flipped an optimize flag that nothing consumed.
- Compile the passes into the image (emit-image manifest): fold, inline, types,
then the jolt.passes façade, after jolt.ir.
- compile-eval and build.ss now run jolt.passes/run-passes between analyze and
emit. Off the direct-link path it is a pure const-fold; `jolt build --opt`
turns on inline + flatten + scalar-replace + type inference (it sets
hc-optimize?, which inline-enabled? reads).
- The seed minter (emit-image) stays analyze -> emit, so the seed is built
un-optimized and the self-host fixpoint is unaffected.
build-smoke already exercised --opt; it now actually optimizes and still matches
the release binary's output. Corpus floor and the fixpoint are green.
Round 1 (correctness + dead code):
- Fix duplicate java.util.HashMap registration in host-static.ss: the alist
impl shadowed the hashtable ctor while leaving the hashtable methods bound,
so .keySet/.values/.remove/.clear crashed. Drop the alist version.
- Delete jolt-core/jolt/reader.clj: a 463-line dead duplicate reader, never
required or compiled (the live reader is host/chez/reader.ss) and drifted.
- Remove dead defs: ir/rt + :rt op + unused ir/op; the Janet branch in
clojure.edn/drain-reader; a shadowed first clojure.string/trim-newline;
io.ss jolt-char-array + the reader def-var (both shadowed by natives-array);
concurrency.ss jolt-future-done?*; compile-eval.ss jolt-analyze-emit.
Round 2 (perf + determinism):
- emit-quoted-map-value / quoted sets now emit sorted by emitted text instead
of host-hash order, which isn't stable across Chez versions (jolt-8479).
- jolt-into folds through a transient, so into/vec/mapv/filterv onto a vector
are O(n) instead of O(n^2).
- deps resolve-deps walks its queue with an index cursor (was subvec-per-pop).
- async channel and agent action queues use amortized-O(1) FIFOs; ArrayList is
backed by a growable vector (O(1) add/get) instead of a list.
Thread/yield was a no-op and Thread/interrupted always returned false. Now:
- yield calls libc sched_yield (resolved once via the process symbols), so a
spin loop relinquishes the CPU. Falls back to a zero-length park if the symbol
can't be resolved.
- each OS thread carries an interrupt flag (a box, thread-local). currentThread
returns a handle wrapping the calling thread's flag, so .interrupt from another
thread sets the target's flag. .isInterrupted reads without clearing; the static
Thread/interrupted reads and clears — JVM semantics.
Consolidates the Thread surface: currentThread + the instance methods live in
io.ss (where the handle and its classloader are built), the flag box + yield +
the interrupted static in host-static.ss. Unit cases cover yield, the read/clear
split, and a cross-thread interrupt over a future.
The from-source Chez build failed on expeditor.c needing X11/Xlib.h — the
expression editor's clipboard. Configure with --disable-x11 (not needed in CI)
and bump the cache key. Add a "Compile a binary" section to the README.
The apt chezscheme package ships petite+scheme only — no kernel dev files — so
the standalone-binary gate skipped on CI, leaving the whole jolt build pipeline
and the --opt inference passes uncovered on Linux. Build Chez v10.4.1 from
source (cached) to get libkernel.a + scheme.h, install the libs the kernel links
against, and set the Linux link flags. buildsmoke now runs for real in CI.
The standalone-binary build needs Chez's kernel dev files (libkernel.a,
scheme.h) and a C compiler. A distro chezscheme package ships neither, so the
gate failed on CI (apt installs petite+scheme only). Preflight for the csv dir
and cc and skip cleanly when they're missing — same pattern as certify skipping
without Clojure. Where the toolchain exists (dev machines), it runs as before;
the discovered csv dir is pinned via JOLT_CHEZ_CSV so the build uses exactly it.
The list led with parity (numeric tower, persistent collections, future/agent/
pmap, core.async) framed as divergences. Keep the four real ones — no JVM/Java
interop, no BigDecimal, no STM, the irregex engine — plus the coverage caveat,
and state the parity as parity.
Wire the optimization gate to build mode. inline-enabled? (which gates the
inference + flatten-lets + scalar-replace passes in jolt.passes/run-passes) was
hardwired off, so those passes had never run on Chez at all. host-contract now
exposes a settable hc-optimize? flag; `jolt build --opt` flips it on during app
emission.
Kept off for the default release build for now: the passes are sound by design
(RFC 0005/0006) but unexercised on Chez, so release stays on the proven
var-deref codegen until they're validated against the corpus. --opt is the
opt-in fast path. buildsmoke checks both modes produce the same result.
This does not yet deliver direct call binding — the backend has no direct-link
emission path (every :var call still routes through jolt-invoke/var-deref) and
the inline-ir host stash is still a stub. Those are the remaining stage-4 levers.
Restores the standalone-binary capability the Janet host had. `bin/joltc build
-m NS -o OUT` AOT-compiles an app into a single self-contained executable — the
whole runtime, clojure.core, stdlib and compiler embedded, no Chez install or
jolt source needed at runtime.
Pipeline (host/chez/build.ss, host primitive jolt.host/build-binary driven by
jolt.main's build command): resolve deps, load the entry namespace recording the
app namespaces in dependency order, re-emit each to Scheme, textually inline the
cli.ss runtime load sequence into one flat source + the app + a launcher, then
compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link against
libkernel.a.
Two non-obvious bits: the compile pass runs in a fresh Chez, not the loaded
runtime (regex.ss shadows top-level `error`, which otherwise bakes a broken
reference into the boot); and the launcher installs scheme-start rather than
running -main at top level, since boot top-level forms execute during heap build
before argv is set, so args only reach -main through scheme-start.
Loader: a require of an in-memory namespace with no source file now no-ops, so
AOT'd app namespaces satisfy require in a built binary.
Mode flags (--opt/--dev, default release) are plumbed; the optimization passes
they gate come in a later stage. RFC 0007 has the design. Gated by `make
buildsmoke`.
Rename src/jolt -> stdlib (the runtime-loaded layer; jolt-core stays the
seed-baked layer) and update the loader / emit-image / doc paths. Drop dead
code: the spike/ experiments, the duplicate clojuredocs-export.edn (json moves
to tools/), the Janet-era jolt.http binding, and the orphaned
persistent_vector.clj whose ns/path didn't even match.
Strip porting residue from comments and docstrings across host/chez, jolt-core,
stdlib, tests, and docs: internal issue ids, "Phase N" markers, and the "vs
Janet" historical exposition, leaving present-tense descriptions and the real
JVM-Clojure semantic contrasts. Same pass over the corpus suite labels. The seed
is unchanged (docstrings/comments aren't emitted), so the self-host fixpoint and
corpus are untouched.
Port tools/spec_coverage.py off the dead janet probe to bin/joltc and regenerate
coverage.md; drop the dead :host/janet rule from certify.clj and regenerate the
conformance profile. Add docs/host-interop.md (the JVM shims and how to register
your own host class from a library) and a writing-style note in CLAUDE.md.
Stabilize the four racy concurrency corpus cases (future-cancel and agent
send/send-off): give the future a sleeping body and the agent a slow action, so
cancel reliably catches an in-flight future and deref reliably reads the
pre-update snapshot. They certify deterministically now, so drop their :flaky
allowlist entries and the orphaned legend.
Two nREPL divergences the library shakeout surfaced — both have a real host
mechanism on Chez.
Interrupt (jolt-amzy): Chez's engine timer (set-timer + thread-local
timer-interrupt-handler) is polled at procedure-call / loop back-edges, so a
running computation — even a tight loop — can be aborted from another thread.
concurrency.ss adds jolt.host/{make-interrupt, interrupt!, run-interruptible}: an
interrupt token is a shared box; run-interruptible arms a periodic timer whose
handler escapes (call/cc) when the token is set, throwing {:jolt/interrupted true}.
The eval thread is reused, not abandoned. (A thread blocked in a __collect_safe
foreign call only sees it on return — like the JVM not killing native code.)
Thread-local *ns* (jolt-6rld): chez-current-ns is now a Chez thread-parameter, so
each session worker / future has its own current ns (vars stay global, only the
pointer is per-thread). *ns* reads derive from it (dyn-binding.ss), and a bound
*ns* drives chez-current-ns — so (binding [*ns* the-ns] (load-string code))
resolves against the-ns, and concurrent in-ns across threads don't clobber each
other. Single-threaded behaviour is unchanged. All runtime .ss — no re-mint.
A reify that doesn't implement a given protocol method now dispatches to that
protocol's extended impls over the reify's host tags (e.g. an Object/default
extension) instead of erroring 'No reified method'. This is malli's pattern: it
reifies some protocols and relies on RegexSchema's default for the rest. A method
with neither a reify impl nor a default still errors.
Surfaced running the DB libraries (migratus) on the jolt db library:
- java.sql.SQLException .getNextException / .getStackTrace / .printStackTrace on
jolt throwables (conditions + ex-info) return nil/empty, so a library walking
the exception chain doesn't crash.
- java.net.URL ctor + .getProtocol (file/http), alongside the existing url shim.
- A generic java.lang.ClassLoader: getSystemClassLoader / a thread's
contextClassLoader resolve a named resource against the source roots (the same
model as clojure.java.io/resource) — file: URL or nil. Thread/currentThread.
These are generic host capabilities, not DB-specific.
The jolt-lang/db next.jdbc surface now runs migratus far enough to connect, build
the migrations table, and discover migrations; migratus's remaining dependency is
java.nio.file (FileSystems/Path/PathMatcher glob), a JVM filesystem API kept out
of core.
Part of the java.* host-class gap (jolt-1nnn). String/format delegates to the
core format engine; NumberFormat getInstance/getNumberInstance/getIntegerInstance
group the integer part and honor min/max fraction digits.
- java.text.SimpleDateFormat.parse parses the RFC1123/1036/asctime patterns
(day/month names, 2-digit-year sliding window, tz token), and .format renders
z/Z/X timezone tokens (GMT/+0000/Z) instead of emitting them literally. Date
gains toLocalDate/toLocalDateTime/before/after/equals. Fixes ring.util.time.
- read / read+string work over a host java.io reader (StringReader wrapped in a
PushbackReader): drain, parse one form, push the tail back. Fixes cuerdas
istr / << string interpolation (and selmer's <<), which read embedded forms
from the template via (read pushback-reader).
Shake-out from the conformance-library sweep. Host-side fixes (runtime .ss,
no re-mint) plus one analyzer change (re-minted):
- Exception fidelity: ex-info and host-constructed throwables (RuntimeException.
etc.) now carry their JVM class, so (class e), instance? across the exception
hierarchy, .getMessage, and clojure.test thrown?/thrown-with-msg? all work.
- .getBytes returns a seqable/countable byte-array and honors UTF-16/UTF-32;
String. decodes them. ->bytevector accepts byte-arrays (Base64).
- Universal .getClass / .toString / .indexOf / .lastIndexOf on any value/seq.
- record? uses the host jrec? predicate (the old (get x :jolt/deftype) crashed
on a sorted-map by invoking its comparator).
- extend-protocol to abstract host types (clojure.lang.Fn/IFn/APersistentVector,
java.net.URI) dispatches.
- New host classes: clojure.lang.PersistentQueue, java.util.ArrayList,
java.net.URI, java.io.File / java.util.UUID ctors, Double/Float ctors+statics,
regex instance? Pattern, System/setProperty.
- *assert* / *print-readably* are real settable/bindable vars.
- (symbol "ns/name") splits the namespace at the last slash.
- letfn fn params desugar destructuring (analyzer; re-minted).
unit.edn gains exinfo/hostobj/queue/hostctor/destructure regression rows.
The built-in nREPL stays minimal (clone/describe/eval/load-file/close) but now
composes a middleware stack so a library can add the heavier features (sessions,
interruptible-eval, completion, lookup) without bloating core.
- A middleware is (fn [handler] (fn [request] ...)); request carries :reply (a
thread-safe send that adds id/session) plus the wire fields. List them in
deps.edn :nrepl/middleware (symbols -> a middleware fn or a vector of them);
jolt.nrepl composes them over the built-in handler.
- Public seam: respond, evaluate, register-ops!, new-session, err-msg.
- Per-connection send lock so middleware replying from other threads don't
interleave bytes. describe advertises built-in + registered ops.
deps.clj surfaces :nrepl/middleware; jolt.main passes it to the server. Built-in
behavior unchanged when no middleware is declared. Runtime, no re-mint.
The line REPL was broken (read-line called nil — the __stdin-read-line host seam
the clojure.core *in* reader drives was never implemented on Chez) and didn't
load the project, so (require '[some.lib]) failed. Now:
- __stdin-read-line reads a line from stdin (get-line); read-line / read / the
REPL work.
- repl resolves the project first (deps on the roots, native libs loaded), so
libraries are available — same context a run gets.
- jolt.nrepl: a jolt-native nREPL server (bencode over a loopback jolt.ffi
socket) speaking the real protocol — clone / describe / eval / load-file /
close, with stdout capture, :ns-scoped eval (in-ns; binding *ns* doesn't drive
load-string resolution here), and real error text. 'joltc nrepl [port]' applies
the project then serves; writes .nrepl-port. Editors (CIDER/Calva/Cursive)
connect and develop live; project libraries load in the session.
- ex-message returns nil for raw Chez conditions, so jolt.host/condition-message
exposes the condition text; the REPL and nREPL surface it instead of an opaque
#<compound condition>.
Why native, not real nREPL: nrepl.server is welded to java.util.concurrent
executors, two compiled Java helper classes, a DynamicClassLoader, Compiler
internals and a JVMTI agent — not faithfully shimmable. The wire protocol, which
is what clients depend on, is small and implemented directly.
Runtime .ss + jolt-core, no re-mint. Full gate green.
jolt-lang/http-client (clj-http-lite over jolt.ffi) replaces it, the same way
ring-janet-adapter replaced the built-in HTTP server. An HTTP client is a library
concern — jolt core no longer ships one or shells out to curl. Apps depend on the
library; (require '[jolt.http-client]) now resolves to its source.
Full gate green.
Gaps the clj-http-lite suite hit running over jolt-lang/http-client:
- with-open now closes a tagged-table stream shim via its .close method (was a
:close field lookup that only matched the Janet-era tables).
- slurp accepts a bytevector / jolt byte-array / a byte input-stream shim and
honors :encoding — clj-http-lite slurps response bodies and :as :stream bodies.
- (.getBytes s charset) and Charset/forName respect the charset (ISO-8859-1 etc.,
one byte per char), not always UTF-8; Charset/forName returns the name string.
- (class e) reads the JVM :class off a throwable tagged-table, so clojure.test's
(thrown? Class …) / (= Class (class e)) match a library's typed exceptions.
- Registered the HTTP/IO exception class names (IOException, UnknownHostException,
SocketTimeoutException, SSLException, …) so the FQ literals self-evaluate.
All runtime .ss shims, no re-mint. Full gate green.
clj-http-lite drives bytes through clojure.java.io/copy (response body into a
ByteArrayOutputStream, request body out) and resolves URLs via io/as-url. Neither
worked for a library shim:
- io/copy was absent. Add it: raw bytes / string / a jhost reader write in one
shot; any other source drains via .read into a buffer and .write to the dest,
both through method dispatch — so a library's tagged-table streams copy without
the host knowing their layout.
- io/as-url ignored a library-registered URL class, so it and (URL. spec)
disagreed (the file-only jhost has no getProtocol/getHost/...). It now honors a
registered URL ctor, falling back to the jhost.
Runtime .ss shims, no re-mint.
The host carries bytes two ways: Chez bytevectors (what String/.getBytes
produce) and jolt byte-arrays (what byte-array / the Java-array shims use). They
didn't interconvert, so code mixing the two — like clj-http-lite, which buffers
into (byte-array n) but encodes via .getBytes and decodes via (String. ^[B body
charset) — broke.
- byte-array now also accepts a bytevector or a string (UTF-8 bytes), so the two
representations convert freely at interop seams.
- (String. bytes [charset]) decodes a bytevector OR a jolt byte-array with the
named charset (UTF-8 default; ISO-8859-1/latin1/ascii = one byte/char). It
previously only took a bytevector and ignored the charset.
Runtime .ss shims, no re-mint. Unit covers both directions + charset.
read-bytes/write-bytes go through UTF-8 (with a latin1 fallback), which mangles
arbitrary binary — gzip payloads, TLS records, any non-text body. An HTTP client
moving bytes between jolt byte-arrays and foreign socket/zlib/OpenSSL buffers
needs byte-exact transfer. read-array returns a fresh byte-array of n bytes from
foreign memory; write-array copies a byte-array's bytes into a pointer. Test
covers a round-trip preserving high bytes (200, 255).
clj-http-lite drives java.net URL/HttpURLConnection and java.io byte streams
through .method interop. The Chez host had __register-class-ctor!/-statics! (what
router/reitit needs) but no way to register instance methods on a shim object or
to extend instance?. Add both, plus jolt.host/table?:
- tagged-table .method dispatch: an htable-arm on record-method-dispatch routes
(.m obj a*) through a per-tag method registry keyed off the table's jolt/type;
unregistered methods fall through (sorted colls are htables too).
- __register-instance-check! installs (fn [class-name val] -> true|false|nil),
nil = fall through; chained ahead of the base instance-check.
Runtime .ss shims, no re-mint. Unit covers dispatch, args, instance? both ways.
An FFI library declares the system shared objects it binds in its deps.edn
(:jolt/native), with per-platform candidate sonames, :optional for feature-gated
deps, and :process for libraries that use the running process's own symbols
(libc sockets). jolt.deps collects them transitively; jolt loads them before the
library's namespaces are required, so foreign-fn bindings resolve — and a missing
required lib fails early with a clear message instead of a cryptic symbol error.
Replaces hardcoded soname-probing inside library .clj files.
The HTTP server moves out of the host into the jolt-lang/ring-janet-adapter
library, which binds sockets itself via jolt.ffi and shuts down cleanly. Drop
host/chez/http-server.ss and the obsolete ffi-server-test (the FFI collect-safe
path is covered by ffi-binding-test; the server by the adapter's own CI).
Derive os.name from Chez's machine-type (*osx -> Mac OS X, else Linux/Windows).
OS-branching code (socket sockaddr layout, etc.) needs the truth; a library
binding sockets via jolt.ffi reads os.name to pick the platform struct layout.
A library binding a blocking native call (accept/recv/connect/...) needs it
emitted __collect_safe so the thread deactivates for the call and doesn't pin
the stop-the-world collector. foreign-fn / defcfn take an optional trailing
:blocking; the backend emits (foreign-procedure __collect_safe ...). Needed for
the ring-janet-adapter socket-server port. ffi-binding-test asserts a thread
parked in a :blocking call doesn't block (collect).
The sqlite/jdbc functionality moves out of the host into the jolt-lang/db
library, which binds libsqlite3 (and libpq) itself via jolt.ffi. A baked
built-in jdbc.core would shadow the library's, so it's removed here. ring-app
gets jdbc.core from the db git dep instead.
A jolt library can now bind its own native dependencies and expose a Clojure API
over them — no jolt built-in required. This is the foundation for moving the
http-client / db / adapter functionality out of the host and into real libraries.
- jolt.ffi/foreign-fn (sugar: defcfn) is a compiler special form: a compile-time
-typed C signature lowers to a real Chez foreign-procedure (analyzer :ffi-fn ->
backend foreign-procedure), so calls are typed and marshaled, not eval'd.
- host/chez/ffi.ss provides the rest under jolt.ffi: load-library, alloc/free,
read/write/sizeof, ptr<->string, null/null?. Loaded after the loader snapshot
so a library's (require '[jolt.ffi]) still loads the macro side.
- Types: int/uint/long/ulong/int64/uint64/size_t/ssize_t/iptr/uptr/double/float/
pointer/string/void/uint8/char.
Validated end to end: a pure-Clojure file binds libc (getpid/strlen/abs) and
libsqlite3 (open/prepare/step/column/finalize over out-param pointers) and runs a
query. Gate test test/chez/ffi-binding-test.ss (make ffi); selfhost holds.
Three related namespace-resolution fixes surfaced porting the clojure.tools.logging
library, all general:
- chez-register-spec! treats a :use :only vector like :require :refer, so a
(:use [ns :only [names]]) clause actually brings those names in. Before, a bare
reference to an :only'd name resolved to nil.
- syntax-quote (hc-sq-symbol) qualifies a referred name to its SOURCE namespace,
not the compile namespace — so a macro that syntax-quotes a referred var (e.g.
clojure.tools.logging/spy reaching clojure.pprint's pprint) expands correctly.
- resolve consults :as aliases and :refers like ns-resolve does; (resolve
'alias/name) was returning nil because the alias wasn't expanded.
Bring the docs in line with the actual implementation now that Chez is the sole
substrate.
Deleted the migration/spike/handoff artifacts that only documented the Janet
era or the port effort: the port plan, phase-0 and foundational-runtime spike
writeups (+ the stray root-level copy), the self-hosting design notes, the
architecture-refactor plan, and spike/chez/RESULTS.md.
Rewrote the current reference docs against the Chez facts: building-and-deps and
tools-deps (no jpm/build step — bin/joltc off the checked-in seed, deps via
jolt.deps into ~/.jolt/gitlibs), libraries (SQLite is built-in jdbc.core over
libsqlite3, not a Janet driver), the conformance/spec test-flow docs (the Chez
corpus runner + certify, no .janet harnesses), and the transient / type-hint /
seed-overlay design notes (Chez representations: mutable transients, flat
copy-on-write vectors, HAMT maps, the seed/overlay twin). Fixed the README
collections line (vectors aren't 32-way tries) and added the ffi/transient gate
targets. rfc 0001's numerics open-question is resolved (the Scheme tower).
Renamed the built-in HTTP adapter to jolt.http.server only (dropped the
ring-janet.adapter alias — a Janet-era name).
The Chez port had landed transients as copy-on-write — each conj!/assoc!/etc.
rebuilt the whole persistent collection. Semantics were right but a transient
vector was O(n^2) to build (the persistent vector is a flat array, so every
conj! copied it); maps/sets were ~O(n log n) since the HAMT only path-copies.
This restores the Janet host's approach: true mutable backing, snapshot once on
persistent!.
vec : a growable Scheme vector (capacity + fill count); conj!/pop! amortized
O(1), persistent! hands off the buffer (exact fit) or trims once.
map : a Chez hashtable keyed by key-hash/jolt= (value equality, nil-safe);
persistent! folds it into a pmap.
set : a Chez hashtable; persistent! folds into a pset.
cow : fallback for anything else (e.g. a sorted coll) keeps the old
copy-on-write path, preserving jolt's superset.
get/count/contains?/nth see through each representation. Building a 400k vector
went from minutes (quadratic) to ~50ms (linear). assoc! keeps the variadic
dangling-key nil-pad on both vectors and maps. test/chez/transient-test.ss pins
the invariants and the linear-time property; wired in as `make transient`.
Two thread-safety bugs in the native FFI layer.
The HTTP server's accept/recv/send were plain foreign-procedures. A thread
inside a foreign call stays active for the stop-the-world collector, so the
accept loop sitting idle in accept() froze GC for the whole process whenever
another thread (a future, an async block) allocated. Mark the three blocking
calls __collect_safe so the thread deactivates for the call's duration —
collection proceeds while the accept thread waits. The args are an fd and
foreign-alloc'd buffers (outside the Scheme heap), so a collection mid-call has
nothing to move.
jolt.http-client built its -D header-file path from an unguarded (set! counter
(+ counter 1)) and counter mod 90000, with no per-process component. Concurrent
requests could compute the same path and clobber each other's headers. Use a
mutex-guarded monotonic counter plus the pid.
test/chez/ffi-server-test.ss exercises both (a (collect) while the server is
idle in accept(), temp-path uniqueness across threads, and a live request) and
is wired into the gate as `make ffi`.
malli loads and validates now. Three divergences surfaced building its registry
and :map schema:
dot-forms: (.iterator coll) on a map fell into the map-as-object branch and was
mis-read as a missing :iterator key (nil), so malli's -vmap got nil and crashed
on .hasNext. Route iterator through the collection-interop path — a jiterator
over the seq (the entry iterator for a map).
host-static: register clojure.lang.LazilyPersistentVector/createOwning (-vmap
fills an object-array then hands it over) and PersistentArrayMap/createWithCheck
(malli's eager entry parser relies on its duplicate-key throw; a missing class
was caught and mis-reported as ::duplicate-keys on every map schema).
Shaking the ring-app example's real library stack out against jolt surfaced a
batch of divergences from JVM Clojure, the biggest being evaluation order.
backend_scheme: call and recur arguments were emitted as bare Scheme operands,
so Chez's unspecified (right-to-left) order won out. Clojure evaluates left to
right, which selmer's reader loop relies on: (recur (add-node ... rdr) (read-char
rdr)) consumed a char early and dropped the first chars of every {{tag}}. Bind
operands to fresh temps in a let* (only when two or more can have side effects,
so hot calls over locals/consts stay un-wrapped). emit-ordered already did this
for collection literals; generalize it.
host-contract: syntax-quote now resolves the alias part of a qualified symbol
(impl/foo -> clojure.tools.logging.impl/foo) instead of leaving it bare, which
limped along via short-name matching until two loaded namespaces (reitit.impl,
clojure.tools.logging.impl) shared the short name and it broke.
collections: key-hash masks with bitwise-and, not fxand — jolt-hash is set!-
decorated per type (records return their own hash) and Chez's equal-hash can be a
bignum, so a key's hash isn't always a fixnum.
seq: even?/odd? handle bignums (JVM accepts any integer; the fxand crashed).
records: Keyword/Symbol .sym/.getName/.toString (honeysql's :clj branch reads
(.sym k)); Throwable .getMessage/.toString over a Chez condition.
host-static: __register-class-ctor!/__register-class-statics! so a host shim
(reitit.trie-jolt) can mirror a Java class.
natives-str: String.intern returns the string.
sqlite: jdbc.core fetch/fetch-one kebab-case column keys (the jolt-lang/db
convention; created_at -> :created-at).
io: a relative io/file path resolves against JOLT_PWD (the user's cwd), not the
repo root the launcher cd'd to — matches JVM cwd semantics, so config.edn loads.
cli: render an uncaught jolt throw (ex-info message + ex-data, or a condition)
instead of Chez's opaque "non-condition value" dump.
Gaps surfaced loading ring-core / hiccup / config / tools.logging on Chez:
- clojure.java.io/resource — resolve a named resource against the loader's
source roots (no classpath), returning a slurp-able File.
- (Object.) constructor — a fresh distinct value (lock / unique sentinel).
- *out* / *err* as dynamic vars over a port-writer, so (binding [*out* *err*] …)
and #'*out* compile and run (tools.logging, selmer).
- :refer :all now registers a refer-all relation, so an unqualified var from a
(require '[ns :refer :all]) resolves at compile time — including #'var.
- java.lang.Character interop ((.toString \+) etc.).
- StringReader accepts a char[] ((StringReader. (char-array s))), not just a
String.
- the \p{L} translation stops just below the UTF-16 surrogate gap (\x{D7FF})
instead of \x{10FFFF} — a range across the surrogates made irregex's char-set
construction call integer->char on a surrogate and crash.
sqlite.ss: jolt.sqlite + jdbc.core (jolt-lang/db's API) over the system
libsqlite3 — open/close, exec, a prepared query returning row maps, text/int/
double parameter binding, last_insert_rowid. The sqlite3 C API is non-variadic
so it binds directly.
http-server.ss: a minimal HTTP/1.1 server over BSD sockets (socket/bind/listen/
accept/recv/send via FFI), one connection at a time on a background accept
thread, synchronous Ring handlers. Parses the request line + headers + a
Content-Length body into a Ring request map (:body a StringReader), formats a
Ring response map back. Exposed as jolt.http.server and, for the example, as
ring-janet.adapter/run-server. macOS and Linux socket-option constants handled.
A synchronous HTTP client def-var!'d into jolt.http-client (get/post/put/delete/
head/request -> {:status :headers :body}), with :headers, :body, :query-params,
:content-type and :insecure?. It shells out to the system `curl` rather than a
direct libcurl FFI: on Apple Silicon curl_easy_setopt is variadic and Chez's
fixed-signature foreign-procedure can't place the value arg on the stack where
the ABI expects it, so a direct bind silently drops the option. curl gives the
same native TLS/redirect/gzip with no per-platform C shim.
format now honours width and the -/0 flags (%-30s, %5d, %05d), not just %.Nf
precision — it was emitting the directive literally.
A minimal truecolor PNG encoder over Chez bytevectors — CRC-32 / Adler-32
framing with DEFLATE "stored" blocks, so there's no compressor to carry. Restores
the jolt.png built-in (image/put!/write) the old host provided; def-var!'d into
the jolt.png namespace and loaded in the CLI before the loader's baked-namespace
snapshot, so (require '[jolt.png]) resolves with no source file. Output verified
to decode as a valid PNG (signature, IHDR, CRC-correct chunks, inflatable IDAT).
A mutable Map keyed by jolt values (jolt-hash/=) with put/get/putAll/containsKey/
size/remove/keySet/values/entrySet — enough for libraries that build a fast
lookup table (malli's fast-registry doto's a HashMap then .get's it). The
"No method M for value" dot-dispatch error now includes the value, which makes
a wrong-type interop target far easier to pin down.
Reader / loader:
- #?@ splicing reader conditionals now actually splice the matched collection's
items into the enclosing sequence; the splice flag was read but ignored, so a
binding vector like [a #?@(:clj [b (.foo b)])] lost its alignment.
- the file loader reads by position and skips a top-level form that reads as
nothing (a :cljs-only #?, a #_ discard, a trailing comment) instead of
treating it as EOF — which silently dropped the rest of a large .cljc file.
- jolt's reader feature set now includes :clj (was {:jolt :default}). jolt is a
Clojure/JVM-compatible host that emulates clojure.lang.* and java.* interop,
so it reads the :clj branch of a .cljc library, not :cljs. This also lets four
more reader-conditional corpus cases pass (floor 2726 -> 2730).
Backend:
- munge-name escapes ' (prime) -> _PRIME_; a Clojure symbol like f' otherwise
emitted a bare ' into Scheme, which is the quote reader macro and unbalanced
the output.
Host shims:
- clojure.java.io/writer (pass through a StringWriter, file-back a path) and a
readLine on the string reader, so line-seq over (io/reader …) works (markdown).
A better "unsupported destructuring pattern: <pat>" error message.
joltc grew from a single -e expression into a real project runner. require now
loads a namespace's .clj/.cljc from the source roots transitively (load-once),
so a multi-file project works; the corpus/unit gates load compile-eval.ss but
not the loader, so their alias-only require is unchanged.
jolt.deps resolves a deps.edn into ordered source roots — git + local deps only
(no Maven), breadth-first so a top-level pin wins, with aliases (:extra-paths/
:extra-deps/:main-opts) and tasks. Git deps clone into a sha-immutable cache
($JOLT_GITLIBS, else ~/.jolt/gitlibs) by shelling out to git via a new
jolt.host/sh primitive. jolt.main dispatches run -m / -M:alias / -A / repl /
path / a deps.edn task. The launcher passes the user's cwd as JOLT_PWD (the
project dir) since it cd's to the repo root for the runtime's relative loads.
Four runtime/reader gaps that blocked real libraries (hiccup, commonmark):
- reader: a type hint on a code form (^String (to-str x)) was lowered to a
runtime (with-meta (to-str x) {:tag String}), mis-applying the hint to the
call's RESULT and throwing when it's a string/number. A :tag hint on an
evaluated form is compile-time only in Clojure — attach it to the form
instead. Collection literals (^:foo [1 2 3]) still get runtime metadata.
- deftype: register the ctor globally by simple class name (like StringBuilder)
so (Name. ...) interop resolves ns-agnostically. host-new resolved the ctor
against the runtime current ns, which is the caller's, not the defining ns,
once a deftype is used across files.
- protocol dispatch: canonical-host-tag now strips the clojure.lang. prefix too
(clojure.lang.Keyword -> Keyword), and keywords/symbols carry the Named tag,
numbers a Ratio tag. An (extend-protocol P clojure.lang.Keyword ...) was
missing the dispatch, so e.g. hiccup rendered <:html> instead of <html>.
- regex: parse leading Java inline flags ((?s)/(?i)/(?m)) and pass the
equivalent irregex options (single-line/case-insensitive/multi-line); irregex
rejects the inline syntax. Adds a java.util.Iterator shim ((.iterator coll)/
.hasNext/.next) for the run!-style loop hiccup compiles.
The two future-cancel cases — (future-cancel (future 1)) and the
future-cancelled? variant — depend on whether future-cancel catches a
trivial future in-flight, which is pure thread-scheduling luck. They were
allowlisted but still counted toward `pass` whenever the race resolved
favorably, so the floor (2728) silently assumed both passed. On a fast dev
machine they always pass; on CI's loaded shared runner one races to a
divergence, dropping pass to 2727 and failing the gate.
Skip them like the undelivered-promise case (neither pass nor fail) so the
race can't perturb the count. Floor drops to the deterministic 2726.
deftype fields tagged ^:unsynchronized-mutable / ^:volatile-mutable can now be
reassigned in place from a method, as on the JVM. A jrec stores fields as cons
cells, so a new jolt-set-field! mutates the pair with set-cdr!. The deftype macro
rewrites (set! mutable-field v) in a method body to (set! (.-field inst) v), and
the analyzer compiles a (set! (.-field obj) v) target to jolt-set-field! — so
both the rewritten symbol form and an explicit interop (set! (.-root this) v) go
through one path. Field reads remain a snapshot at method entry, which is correct
for the universal read-then-set pattern (a repeated set! of the same field in one
call would read the entry value).
Closes the set!-of-local SCI failures: SCI load 202 -> 205/218.
Bring the formal definition in line with this session's language work:
- grammar.ebnf: numbers are a real tower (exact integer / Ratio / double); the M
suffix reads a real BigDecimal, N an exact integer (drop the stale Janet note).
- 02-reader S5: M is a real java.math.BigDecimal with scale-insensitive equality.
- 03-special-forms: document the read -> macroexpand -> analyze order (macros
expand before special-form dispatch); special-form heads are not shadowable but
macros are and value-position locals may be named like a special; set! on a var
sets the innermost binding (else root); letfn is a primitive with letrec*
semantics.
Found in a read/eval review: a local named like a special form wrongly took
over operator position. (let [if (fn ...)] (if true 1 2)) returned the fn, but
per spec section 3 (and the reference) special-form heads are not shadowable;
only macros are. Two fixes: drop the (not shadowed) guard on the special-form
branch of analyze-list (so an (if ...) head is always the special), and prefix
a local whose name is a Scheme keyword when emitting (so a value local legally
named if does not shadow the (if ...) the back end emits). Value-position
locals named if/or/case still work.
The analyzer checked special forms before expanding macros, the reverse of the
canonical read -> macroexpand -> analyze order (Clojure/CLJS analyze-seq). Move
macroexpansion to the front of analyze-list. Knock-on fixes:
- letfn was both a (broken) macro expanding to let* AND a primitive special
(analyze-letfn, proper letrec*). Macroexpand-first surfaced the macro, breaking
mutual recursion; remove the macro, keep letfn a primitive.
- defmacro is now compiled by the analyzer (a :set-var-style :defmacro node that
defs the expander fn via the fn macro — so destructuring arglists desugar — and
marks the var a macro), so a non-top-level (when … (defmacro …)) works. The
runtime spine's separate top-level defmacro interception is removed: one path.
SCI load 162 -> 202/218.
The analyzer punted set! as uncompilable. Add it as a special form: (set! sym
val) on a var emits jolt-var-set, which updates the innermost thread binding (or
the root when unbound), returning val. A local target (deftype mutable field) or
an interop (.-field) target stays uncompilable for now. Also defines
*warn-on-reflection* (false) so set! on it resolves. SCI load 186 -> 196/218.
Chez rejects duplicate lambda formals, so any (fn [_ _] ...) failed to compile
— including every macro expander, whose &form/&env slots both expand to _. The
analyzer now renames each earlier duplicate param to a fresh name (Clojure binds
the last occurrence, so the earlier ones are shadowed and unreferenceable). SCI
load 162 -> 186/218.
A vector's seq is now a real chunked-seq (chunked-seq? true), matching Clojure/
CLJS. Each vector-seq cell carries its backing vector + element index as two
cseq fields (cvec/ci, no extra allocation vs the old lazy cell), so:
- chunk-first hands out a 32-element block (a pvec slice), chunk-rest is the
seq at the next block boundary — the ChunkedSeq contract (chunk-first ++
chunk-rest == the seq);
- reduce/transduce take a fast path that walks the backing vector by index in
a tight loop with no per-element seq cells (reduce over a 1M-vector ~0.4s).
The seq cell stays a cseq, so first/rest/count/printing and the ~26 cseq?
dispatch sites are untouched. The eager chunk-buffer model (chunk-buffer/chunk/
chunk-cons) is preserved for the round-trip case. No seed change (runtime only).
(type r) returned a symbol user.TyR, so (= (symbol (str (type r))) (type r))
was true; the JVM's type is a Class (not a Symbol) so it's false. jolt models
classes as strings, so a record's type is now its ns-qualified class-name
string — equal to (class r), as on the JVM where type and class coincide for a
record. The symbol-keyed print-method defmethods already fall through to the
default record printing, so they're unaffected. Closes type-of-record.
bigdec / 1.5M / 0.0M silently produced doubles. Add a jbigdec value type
{unscaled, scale} over Chez exact integers (host/chez/bigdec.ss): value =
unscaled * 10^-scale. An M-suffix literal reads to a :bigdec form that the back
end lowers to jolt-bigdec-from-string (same IR-leaf path as #inst/#uuid); bigdec
coerces a number/string. Equality is by value (1.0M = 1.00M true, 3M = 3 false),
str drops the M and pr keeps it, class is java.math.BigDecimal, decimal? is true.
Arithmetic contagion isn't modelled (out of scope). The old corpus cases passed
spuriously as doubles; they now exercise a genuine BigDecimal.
(def ^String tv ...) left (:tag (meta (var tv))) as the unresolved "String";
the JVM compiler resolves the hint to java.lang.String at def time. Add a
resolve-class-hint host seam (built from the existing class-token table) and
resolve a def's :tag through it in the analyzer. The reader path
(read-string "^String x") stays unresolved, matching the JVM (only the
compiler resolves). Closes ^Type-tag-on-var.
(map? *in*) was true because *in* was a plain map of read-line-fn/read-fn
closures; the JVM *in* is a java.io.Reader so map? is false. A defrecord
doesn't help (records are maps). Make the reader a reify over a new IReader
protocol — a non-map value — and route read/read-line/read+string/line-seq
through its -read-line/-read-form/-read+string methods instead of keyword
access. with-in-str's __string-reader and the stdin *in* both reify it.
Closes *in*-bound + *in*-is-bound.
A macro like (defmacro cur-ns [] `(str ~*ns*)) splices the live *ns* value
into its expansion, leaving an opaque jns object as a list element. The
analyzer had no way to carry a runtime value and threw uncompilable — the last
remaining corpus crash. Recognize a jns via the host contract (form-ns-value?)
and emit a :the-ns leaf that reconstructs it by name (intern-ns!) at the call
site, the same IR-leaf pattern as regex/inst/uuid. Closes unquote-*ns*-in-
template; corpus crash count -> 0.
A namespace fast path rather than a general constant pool: it's the only
embedded-value case in the corpus and the common real-world one (libs splice
~*ns*). A general pool can come later if other value types appear.
deftype/defrecord inline protocol methods went through extend-type ->
register-method, so a record implementing a protocol inline showed up in
(extenders P) — the JVM only lists extend/extend-type/extend-protocol
registrations there (inline impls compile into the class). Add
register-inline-method: it registers for dispatch under the record tag but
skips the extender mark. The mark lives inside type-registry so the per-case
corpus prune restores it. Closes corpus lists-extended-type + seq-of-tags.
The defmethod/print-method record cases, symbol-hint, and source-order entries no
longer diverge (closed by the defmethod-setup + earlier fixes); drop them. 0 new
divergences, corpus 2720/2741, 20 genuine gaps remain.
definterface now expands to (do (def name {}) 'name) so (var? (definterface ...))
is false, matching the JVM where it yields the interface Class. ns-imports returns
the 96 auto-imported java.lang classes (short symbol -> canonical name) so
(count (ns-imports 'user)) is 96. Re-minted for the macro change. Corpus 2718->2720.
class number/string/keyword/name + atom?/instance? Atom pass after the class
refinements; drop their stale allowlist entries. Corpus 2705/2741 0 new div.
The prior fix resolved an unqualified defmethod to clojure.core's multifn, which
broke SCI (it relies on per-ns shadow multimethods — hung loading core_protocols).
Keep the shadow, but when auto-creating it copy the dispatch fn + default from a
same-named clojure.core multifn (e.g. print-method's 2-arg dispatch) instead of
the 1-arg identity that crashed (print-method x w). Also trim the FQN class
tokens to value classes only (the collection interfaces shadowed names SCI uses).
Corpus 2705/2741 0 new div; SCI 162/218 restored; cross-ns + direct print-method
overrides work.
- (class x) returns per-type JVM class names (Long/Double/Ratio/Character/Atom),
not a blanket java.lang.Number.
- register fully-qualified class tokens (java.lang.Long, clojure.lang.Keyword,
clojure.lang.Atom, ...) that self-evaluate to their name, so (= (class 1)
java.lang.Long) and (instance? clojure.lang.Atom x) resolve.
- instance? recognizes Long/Double/Ratio/Character/Symbol/Atom/IFn built-ins.
Closes class number/string/keyword/name, instance? Atom, atom?. Corpus 2699->2705.
(defmethod print-method ...) from the user ns auto-created a stray user/print-method
with identity dispatch -> 'incorrect number of arguments 2' on (print-method x w).
Resolve an unqualified multifn name through current-ns -> :refer -> clojure.core
(like var resolution) before auto-creating. Fixes direct print-method/print-dup
override calls; pairs with the cross-ns defmethod fix.
Allowlist review found three addressable divergences:
- unchecked-char returned a number; the JVM returns a char.
- the readable printer (pr-str, coll elements, the -e/REPL printer) rendered
infinities as Infinity/inf; Clojure's readable form is ##Inf/##-Inf/##NaN
(str/print still gives Infinity). So (pr-str ##Inf) => ##Inf, (str [##Inf]) =>
[##Inf], (str ##Inf) => Infinity.
Corpus 2695->2698; allowlist 43->40 (drop the 3 now-passing entries). Re-minted.
A macro that syntax-quoted interop — `(.. (StringBuilder.) (.append x)) — had
its .method / Class. / .-field heads qualified to the compile ns (user/.append,
user/StringBuilder.), so they read as 'Unknown class user' at expansion. Like
Clojure, leave interop-head symbols bare in syntax-quote. Fixes any macro
templating interop, not just the one corpus case. Corpus 2694->2695.
(reduce f init (reify clojure.lang.IReduceInit (reduce [_ f i] ...))) tried to
seq the reify and threw 'not seqable'. When the coll is a reify carrying a reduce
method, drive the reduction through it. Corpus 2693->2694.
The . special form rejected a non-symbol member; a keyword member now lowers to
an invoke of the keyword on the target ((. {:value 41} :value) => 41, as on the
JVM). Added a form-keyword? contract seam. Corpus 2692->2693. Re-minted.
(:import [other.ns Type]) was a no-op (import unbound), so (Type. ...) failed
with 'Unknown class'. Bind import to register each named type's ctor closure
under the current ns. Corpus 2691->2692.
(.. x m ...) failed: the analyzer classified the .. head as a .method interop
call (method-head? matched any "."-prefixed name) and form-special?
(hc-interop-head?) also flagged it, so it never reached the macro check. Exclude
".." from both (the char after "." being "." means the threading macro, not
.method). Corpus 2690->2691. Re-minted.
JVM assoc! is variadic: with a complete first pair present (>=3 kvs), a trailing
lone key fills nil ((assoc! t :a 1 :b) => {:a 1 :b nil}); a lone key alone (1 kv)
is still a wrong-arity throw. jolt delegated to the strict persistent assoc which
threw on any odd count. Pad a trailing nil for odd kvs >=3. Corpus 2688->2690.
The per-form eval passed a FIXED compile-ns to every subform of a top-level do,
so a runtime (ns ...)/(in-ns ...) didn't redirect later defs/refs — defs landed
in the wrong ns and qualified refs hit host-static ("Unknown class"). Thread
the current ns: each subform analyzes in (chez-current-ns), which ns/in-ns move.
That exposed two more gaps, now fixed:
- use refers ALL of a target's public vars (a refer-all table consulted by
chez-resolve-refer) — was bound to plain require (explicit :refer only).
- defmethod on a QUALIFIED multifn (cf.mm/ext from another ns) resolves in the
symbol's ns, not the current one (was auto-creating a stray multifn).
Corpus 2684->2688, 0 new divergences; floor raised. No re-mint (runtime shims).
sorted-map seq/first/entries built plain [k v] vectors, so map-entry? was false
and key/val threw. Build them via a new jolt.host/map-entry seam (entry-flagged
pvec), matching a regular map's entries. Re-minted.
The self-host byte-fixpoint (make selfhost) only holds on the Chez that minted
the seed — CI's Debian Chez emits byte-different output for some constructs
(isolated to the dedupe re-mint), so it failed there. The checked-in seed RUNS
correctly on any Chez, so CI now runs 'make ci' (corpus/unit/smoke/sci/certify);
'make test' keeps selfhost for local dev. Cross-version emit determinism tracked
in jolt-8479.
(into [] (dedupe) coll) / (sequence (dedupe) coll) threw an arity error — dedupe
was [coll]-only. Add the 0-arg stateful transducer (tracks [seen? prev] in a
volatile, no sentinel). Re-minted.
- == 1-arg returns true for any value (Clojure short-circuits before the number
check), not 'requires numbers'.
- current-time-ms wired to now-millis so the time macro works.
- subvec truncates float/ratio indices via long (Scheme quotient rejects flonums).
- defonce checks bound? not var-get — in a top-level do the name is already an
unbound interned cell, which var-get throws on.
- drop the line-seq corpus row (used janet/spit, N/A); allowlist char-array
(needs Class/forName "[C").
Corpus 2678->2683, floor raised. Re-minted. Full gate green; CI green.
jolt-cf1q.7
Chez derives its boot-file name from argv0, so a symlink named chez looks for a
nonexistent chez.boot (CI failed at the first gate step). Replace the symlink
with a wrapper that exec's scheme, preserving argv0 so the boot files resolve.
Remove the 17 rows that exercised the Janet FFI bridge (interop/janet bridge,
interop/jolt.interop) and the Janet build-time env scrub (host-interop/bake env
scrub, janet.os/setenv) — none exist on any non-Janet host, so they only added
crash noise. Portable interop is covered elsewhere. corpus 2920->2903 rows;
gate 2678/2742 0 new div, certify 0 new/0 stale.
Re-port the SCI compatibility stress test to joltc: host/chez/run-sci.ss loads
borkdude/sci's own source (vendor/sci, re-vendored) through the spine and
requires its forms to compile+eval. Floor-gated at 160/218 forms (the tail is
genuine host gaps — set! on vars, some macro/def shapes); raise as they close.
Wired into 'make test' (skips if the submodule isn't checked out).
jolt-cf1q.6
Rephrase comments that pointed at deleted Janet files (emit.janet, the seed
sources, 'the Janet back end punts ...') to present-tense descriptions of the
Chez behavior. Comment/docstring-only; the self-host fixpoint is unchanged
(comments don't affect the compiled seed).
Delete five files that were Janet-host shims with no Chez path: clojure.java.io
(provided natively by host/chez/io.ss), and jolt.{nrepl,png,interop,shell}
(the janet.* bridge, os/shell, janet.net — none exist on Chez).
jolt-cf1q.6
Rewrite the README, CLAUDE.md build/architecture sections, test/chez/README,
and conformance SPEC for the Janet-free world: bin/joltc + make test, the
self-hosting bootstrap, the frozen JVM-sourced corpus. CI installs Chez + JDK/
Clojure and runs 'make test' (was Janet/jpm).
jolt-cf1q.6
Remove the Janet seed (src/jolt/*.janet: reader, value layer, vars/ns, the
tree-walking interpreter, the Janet backend, the optimizing compiler), the
Janet->Scheme cross-compiler (host/chez/{driver,emit,jolt-chez}.janet),
bin/jolt-chez, the jpm build (project.janet) and the Janet test runner
(run-tests.janet), plus the entire Janet test suite. jolt now builds and runs
on Chez alone: bin/joltc off the checked-in seed, bootstrap.ss to rebuild it.
The portable Clojure stays: jolt-core/**, host/chez/**.ss, and the stdlib +
tooling under src/jolt/clojure + src/jolt/jolt (read by the seed build, no
Janet). The gate is 'make test' (self-host, corpus, unit, cli smoke, certify).
Drop the sci and clojure-test-suite submodules (used only by deleted Janet
integration tests); irregex stays.
Filesystem corpus/unit cases that probed project.janet now probe README.md.
jolt-cf1q.6
Add a Janet-free gate so correctness can be judged with only Chez + Clojure:
- host/chez/run-corpus.ss: corpus.edn vs JVM expecteds, lifting the per-case ns
isolation from the old Janet driver; reads corpus.edn via the jolt reader.
- host/chez/run-unit.ss + test/chez/unit.edn: the host-specific unit cases,
evaluated in-process and compared to baked expecteds.
- host/chez/selfcheck.sh: self-host fixpoint (bootstrap.ss rebuild == checked-in seed).
- host/chez/smoke.sh: real bin/joltc CLI smoke.
- host/chez/remint.sh: re-mint the seed to a byte-fixpoint after a source change.
- Makefile: 'make test' runs the lot; 'make remint' rebuilds the seed.
Numbers match the Janet gate: corpus 2679/2757 0 new div, unit 450/450, certify
0 new/0 stale.
jolt-cf1q.6
The unit tests in test/chez/_*.janet now drive bin/joltc (the zero-Janet
spine) and judge against baked expected values instead of a live build/jolt
run. Ten of them captured the oracle from build/jolt per case; their values
are now literals (one env-dependent javastatic case became a predicate so it
stays portable). The rest already had literal expecteds with a redundant
build/jolt sanity check, now dropped.
Retire emit-test/emit-parity/reader-parity: they compared the Chez/Clojure
path against a live Janet evaluation, emitter, or reader. That migration check
is done, and run-corpus-zero-janet (Chez analyzer vs the JVM corpus) plus
certify.clj cover correctness now.
Rewrite the README for the current zero-Janet gate.
jolt-5oci
run-corpus.janet drove a target binary (default build/jolt, the Janet host)
through the corpus and run-corpus-chez.janet was the all-flonum subset probe.
Both compare against corpus.edn, which is now JVM-sourced, so the all-flonum
hosts diverge on every numeric/host case. The zero-Janet spine gate
(run-corpus-zero-janet.janet) plus certify.clj are the oracles; drop these and
the stale test/chez/known-divergences.edn allowlist they used.
corpus.edn :expected is now the value reference JVM Clojure produces, set by the
new test/conformance/regen-corpus.clj (one JVM process, per-row thread watchdog).
167 rows moved to the JVM value: ratios (/ 1 2)=>1/2, doubles (double 3)=>3.0,
shared-heap concurrency (the future/pmap/agent cases), clojure.math doubles. The
JVM is the spec; jolt is measured against it.
known-divergences.edn shrinks to the rows whose JVM value is an opaque host object
that can't round-trip to source (Java arrays, transients, atoms, beans, proxies,
chunks all print as #object[..@addr]) plus (fn* foo) and a few racy concurrency
cases (:flaky). The zero-janet gate's allowlist becomes the set of host gaps vs the
JVM spec (no Class/array/BigDecimal, :jolt reader, jolt's own printing).
Math/clojure.math sqrt/pow/floor/trig now return doubles (Chez returns exact for
exact args, e.g. (sqrt 9)=>3); JVM always returns a double.
extract-corpus.janet no longer writes corpus.edn unless asked (the test runner
imported it and was silently overwriting the JVM corpus with the spec sources'
placeholder answers). The prelude parity gate is deleted — the zero-janet spine +
certify.clj are the oracles.
zero-janet 2678 (0 new divergences), certify 0 new / 0 stale, emit-test 330/330.
jolt was all-flonum (one :number type, inherited from Janet whose only number
type is a double). The Chez runtime has a full numeric tower, so the zero-Janet
path now carries it = JVM Clojure semantics:
(/ 1 2) => 1/2 (exact Ratio, was 0.5)
(integer? 3) => true (integer? 3.0) => false (float? 3.0) => true
(ratio? (/ 1 2)) => true (= 3 3.0) => false (== 3 3.0) => true
(+ 1 2) => 3 (exact) (/ 1.0 2) => 0.5 (double)
jolt= was already exactness-aware (values.ss) and == is value-equality, so
=/== match the JVM split. The reader preserves exactness (integer literals exact,
a/b ratios exact rationals, decimals/exponents flonums); backend_scheme emit-const
renders exact ints/ratios and flonums faithfully; the value-position arithmetic,
count, int, compare, bit ops, parseLong, string .length/.indexOf, range,
timestamps, and array bytes return exact integers (= JVM int/long) instead of
coercing to flonum. double/parseDouble/clojure.math floor|ceil|signum stay double.
Only the zero-Janet path carries the tower (the Janet reader loses exactness into
a double before emit). The prelude/all-flonum path is unaffected for compiled code;
the runtime reader is shared, so a couple of all-flonum reader assertions become
value (==) assertions. ~16 numeric corpus cases now give the JVM tower value vs the
Janet-era :expected and are allowlisted as tower divergences (Chez == reference
JVM) pending the corpus flip to JVM (jolt-ecz0). No BigDecimal type (1M).
Re-minted. zero-janet 2682 (floor 2698->2682, the reclassified tower cases), 0 new
divergences; fixpoint 10/10, bootstrap 6/6, spine 35/35, cli 49/49; Janet gate 155
files 0 failed.
Two Chez reader bugs, both JVM-parity gaps:
inc'/+'/foo' (trailing apostrophe) were mis-read as a symbol followed by a
quote macro, because the reader treated ' as a terminator. In Clojure ' is a
NON-terminating macro char (constituent after the first char). Since the seed
is minted on Chez, (def inc' inc) became (def inc 'inc), clobbering inc's var
cell with its own symbol -- so (var-get (var inc)) returned the symbol, not the
fn. Drop ' from the token terminator set; a leading ' still quotes.
^bytes [b] / ^String [x y] return-type hints: the Chez reader lowered ^meta on
a collection to a (with-meta vec meta) form, but emitted a QUALIFIED
clojure.core/with-meta while the Janet reader emits a bare with-meta -- so the
fn/defn macros' unwrap logic (matching the bare head) slipped past it and choked
on a non-vector arglist. Emit bare with-meta to match Janet, and unwrap a
(with-meta <vec> _) arglist in analyze-fn as a backstop.
Re-minted the seed. zero-janet 2699, prelude 2652, Janet gate 155/0, fixpoint
10/10, bootstrap 6/6, all 0 new divergences.
realized? threw 'not supported on' for a jolt-lazyseq record (the overlay
reads :jolt/type); add a jolt-lazyseq? arm to the post-prelude wrapper
reading the record's own realized? flag.
conj! 1-arity (conj! coll) is the transducer-completion arity and returns
coll as-is on the JVM, no transient check — we threw 'not a transient'.
Both gates: zero-janet 2696->2698, prelude 2649->2652, 0 new divergences.
Audit of the janet.*/jolt.interop/STM corpus cases vs Chez equivalents: the Janet
FFI-bridge cases (janet/string, janet/type, janet.math/sqrt, janet.string/ascii-
upper) test functionality already covered by PORTABLE corpus cases (str 29, type 9,
sqrt 3, upper-case 9), so they're safe to delete in Phase 5. Two needed a Chez
equivalent so they pass instead of being lost:
- clojure.lang.LockingTransaction/isRunning -> false (no STM on jolt).
- line-seq: the corpus case used janet/spit setup but was the SOLE line-seq
coverage (0 other cases). Ported janet/spit->spit, and added a native Chez
line-seq (drain a jhost io/reader + split on newline; no trailing empty line)
that delegates a Janet map-reader to the overlay version.
zero-Janet 2694->2696, prelude floor 2648; self-host + Janet gate + JVM cert green.
jolt-cf1q.7 jolt-0obq
hc-list? required cseq-list? (a reader-built list), so a form built at runtime via
concat/map/cons — a lazy cseq with list?=#f — was rejected as "unsupported form".
In Clojure any seq is a valid call form, so accept any cseq. Lights up macros that
build their expansion with concat/list and (eval seq-form).
zero-Janet 2692->2694, 0 new divergences; self-host + Janet gate + JVM cert green.
jolt-cf1q.7
Date/time (inst-time.ss): java.util.Date / java.sql.Timestamp ctors accept ms or
another date value (ms-of) -> a jinst; java.text.SimpleDateFormat (pattern + .format
via the existing format-ms UTC engine; .setTimeZone accepted); java.util.TimeZone/
getTimeZone. instance? answers Date true / Timestamp false for a jinst (a Date is
not a Timestamp on the JVM).
clojure.edn/read over a reader (io.ss + post-prelude): the overlay edn.clj's
drain-reader is janet/type-coupled, so Chez drains the jhost StringReader/
PushbackReader to a string and reads the first EDN form. Unblocks jolt-uicd.
Native Chez throughout (no vendoring): Chez date arithmetic + string ports. zero-Janet
2688->2692, 0 new divergences; self-host + Janet gate + JVM cert green.
jolt-cf1q.7 jolt-dcmm jolt-7t3l jolt-uicd
deftype binds the type name as a VAR (the make-deftype-ctor closure), but (P. 5)
lowers to (host-new "P"), which only checked class-ctors-tbl -> "No constructor".
Fall back to resolving the class name as a var in the current ns / clojure.core
and invoking it — so (P. args) constructs the same jrec as the ->P factory, and
protocol method dispatch (.m / .-field) over it works.
zero-Janet 2685->2688 (no-constructor 5->2); prelude floor bumped to 2641 (the
delay batch's run). Self-host + Janet gates + JVM cert green.
jolt-cf1q.7
Add a thread-safe delay type (concurrency.ss): make-delay wraps a thunk; deref/
force run it once under a lock and cache (JVM delays are thread-safe + memoized).
delay? and realized?-on-a-delay are native; the overlay's `delay` macro
(-> make-delay) and `force` (-> deref) now work. realized? wrapper (post-prelude)
and the deref chain gain a delay arm. Removed the np-delay? stub from
natives-parity.ss (the real type lives in concurrency.ss).
Seed unchanged (no re-mint). zero-Janet 2673->2685, 0 new divergences; Janet gate
+ JVM cert green.
jolt-cf1q.7
The zero-Janet runner wrapped each case as (= EXPECTED ACTUAL) and checked the
result was true. That nests ACTUAL's top-level (do ...), so a case like
(do (defmacro m ...) (m 1)) can't use the macro it just defined — the analyzer
punted defmacro -> "uncompilable" (35 cases).
Match certify.clj's eval-isolated instead: carry EXPECTED and ACTUAL as separate
sources and evaluate ACTUAL as its own top-level program (jolt-compile-eval unrolls
the top-level do, so a macro defined earlier is usable later), then compare to
EXPECTED with =. Evaluating ACTUAL from SOURCE (not (eval (quote A))) preserves the
reader's map-literal source order, so the eval-order cases still pass.
eval-corpus-zero-janet / program-corpus-zero-janet now use a 3-field TSV
(label/expected/actual); run-corpus-zero-janet's per-case debug path evals both
sides too. Only run-corpus-zero-janet uses these (the prelude gate is untouched).
zero-Janet 2642->2673 (analyzer "uncompilable" 35->4); 1 new allowlisted divergence
(`{:a ~x :b ~y} syntax-quote map construction doesn't preserve source eval order —
pmap is unordered). Janet gate + JVM cert green.
jolt-cf1q.7
host/chez/natives-array.ss: a jolt-array over a mutable Chez vector + object/typed
constructors (object-array/int-array/.../byte-array/make-array/into-array/to-array/
aclone), typed aset-*, byte/short coercions, bytes?/bytes/ints/..., and eager
chunk-buffer/chunk-* (Jolt doesn't chunk). count/nth/seq/get and jolt.host/ref-put!
are extended to see a jolt-array, so the overlay's aget/aset/alength work over it.
Numbers it produces are flonums (jolt's rep) so exactness-aware = holds. char-array
stays in io.ss (a char-SEQ that io/reader/str/slurp consume).
natives-parity.ss adds: __reader-features / -set!, reader-conditional, re-matcher,
delay? (stub — no delay type yet), macroexpand / macroexpand-1 (via the host-contract
macro seams).
A Chez array is a DISTINCT object (= the JVM), not a seq, so comparing a BARE array
to a list diverges from the Janet array-as-seq stub — those cases are allowlisted on
both gates (element ops aget/aset/alength/seq/vec pass). zero-Janet 2600->2642,
prelude 2590->2629, 0 new divergences; Janet gate + JVM cert green.
jolt-cf1q.7
Native Chez shims for clojure.core fns that live in the Janet seed but had none
on the zero-Janet spine (they resolved to nil -> "not a fn"):
- host/chez/natives-parity.ss: hash / hash-combine / hash-ordered-coll /
hash-unordered-coll (24-bit masked like core_extra), transient?, rseq (vectors +
sorted colls), cat (transducer).
- jolt-invoke now dispatches a TRANSIENT vec/map/set as a fn (callable on the JVM):
((transient [10 20 30]) 1) -> 20.
- ns.ss: ns-resolve, ns-imports, remove-ns, intern, alias, ns-unalias, refer,
ns-refers, refer-clojure, alter-meta!, reset-meta!, and a real ns-aliases (was a
stub returning {}).
- runtime (require ...)/(use ...) now register :as/:refer into the Chez ns tables
(was a no-op). The Chez analyzer already pre-registers at analyze time, but when
the JANET analyzer compiled the form (prelude path) the Chez tables stayed empty,
so ns-aliases/ns-resolve over an alias diverged — this fixes both paths.
Seed unchanged (overlay doesn't reference these at mint time). zero-Janet corpus
2567->2600, prelude 2557->2590, 0 new divergences on either; Janet gate + JVM cert
green. Filed jolt-vgrp for the pre-existing var-get-of-scalar-native-op quirk.
jolt-cf1q.7
Mirrors test/bench/core-bench.janet's 8 compute programs + methodology (load the
runtime once, time compile+run of each, min of N) so the zero-Janet Chez path is
comparable to the Janet compile path.
Result: Chez is 3-33x faster than Janet across the set (~320ms vs ~5000ms total,
~15x), biggest on seq/map/reduce, smallest on fib (3.2x). So deleting Janet is a
perf win, which satisfies the "perf confirmed" precondition Phase 5 gates on. Chez
is ~2-28x slower than JVM Clojure, expected and not the bar (and the Janet-only
optimizing modes haven't been tried on Chez).
jolt-cf1q.5
Replace the Janet synchronous agent shim (agent = atom, send applies inline) with
JVM-style async agents: send/send-off enqueue an action and a single worker thread
per agent applies them in order; deref reads the current (maybe not-yet-updated)
state without blocking; await blocks until the queue drains. A validator rejection
or a thrown action puts the agent in an error state (agent-error) and halts the
queue; restart-agent clears it. send and send-off share one serialized worker (a
superset of the JVM's fixed/cached pool split). Native versions re-asserted in
post-prelude over the overlay; await/restart-agent are new.
Corpus: the two "send/send-off applies" cases do (send a f) (deref a) with no
await, so they now read state before the action runs — diverging like the JVM
(the suite was literally "synchronous shim"). Allowlisted on both gates; floors
-2 (zero-Janet 2569->2567, prelude 2559->2557). cli-test covers async agents via
await (ordered 100-send dispatch, error capture) — 49/49. Janet gate + JVM cert
green; 0 new divergences on either corpus.
jolt-byjr
No mature Chez fibers library exists and this is a threaded Chez build, so a go
block is an OS thread and a channel is a mutex+condition blocking queue: <! / >!
are the blocking <!! / >!! and work anywhere (no CPS transform), like the Janet
stackful-fiber model but with real parallelism and a shared heap.
host/chez/async.ss provides chan (unbuffered rendezvous / fixed / dropping /
sliding), <! >! <!! >!! close! alts! timeout put! take! buffer ctors, channel
transducers, and go-spawn, all def-var!'d into clojure.core.async; go/go-loop/
thread are macros (mark-macro!) expanding to go-spawn, mirroring src/jolt/
async.janet. Binding conveyance rides the thread-parameter binding stack from
pt.1. alts! polls with a 1ms backoff (no cross-channel wait-set yet) and is
take-only, matching the Janet impl.
(require '[clojure.core.async ...]) resolves it with no file load — the vars are
resident and require just registers the :as/:refer.
cli-test covers go/buffered-drain/nested-<!/alts!/transducer/timeout/binding-
conveyance (43/43). core.async isn't in the conformance corpus (feature-gated
:async/core-async), so coverage is the Chez cli-test plus the existing Janet
core-async-spec. Seed unchanged (no .clj touched). Prelude corpus 2534->2559,
zero-Janet 2569, 0 new divergences on either; Janet gate + JVM cert green.
jolt-byjr
future/future-call run the body on a native thread (fork-thread) over the SAME
heap — JVM semantics, not Janet's isolated-heap snapshot. deref blocks on a
mutex+condition latch; timed (deref f ms val) uses an absolute deadline.
promise is a real blocking promise (deref parks until deliver), replacing the
Janet non-blocking atom shim. future?/future-done?/future-cancelled?/future-cancel
/realized? are native (the overlay versions read Janet map keys); re-asserted in
post-prelude over the overlay. pmap/pcalls/pvalues (overlay, over future) light
up for free.
Thread-safety this forces:
- atoms get a per-atom mutex; swap!/swap-vals! are a JVM-style CAS loop (f runs
outside the lock, so a watch/validator can deref the same atom); reset!/
compare-and-set! are atomic.
- the dynamic binding stack becomes a Chez thread-parameter, so each future/thread
has its own; Chez inherits it at fork, giving binding conveyance (the shim also
installs an explicit snapshot).
- Thread/sleep really sleeps now (a worker sleeping doesn't block the parent).
Re-minted the seed: future-call now resolves at compile time, so pmap compiles to
a var-deref instead of the host-static-call fallback that crashed. image.ss
unchanged.
Corpus: the 2 snapshot cases now match the JVM (shared) not Janet (isolated) —
allowlisted on both Chez gates; the two racy future-cancel cases allowlisted;
"promise undelivered" (blocks on JVM/Chez, profile :bucket :timeout) skipped like
:throws. Zero-Janet corpus 2544 -> 2569, 0 new divergences, floor raised. Full
Janet gate + JVM cert green.
jolt-byjr
The compiler image is already resident at runtime on the Chez spine, so eval
and load-string are just wiring: make them clojure.core functions instead of
analyzer special forms.
- eval / load-string are now functions, not special forms. Dropped "eval" from
the host-contract special-symbol lists so it resolves as an ordinary var, and
def-var! both in compile-eval.ss. eval takes an already-read form (e.g. from
quote/list) and compiles+evals it in the current ns; load-string reads every
form from a source string and evals each, returning the last.
- Runtime defmacro: jolt-compile-eval-form intercepts a (defmacro ...) form
before analysis, defs the expander fn + mark-macro!s the var, exactly as
emit-image.ss does at build time. The two helpers (macro-form? / defmacro->fn)
move to compile-eval.ss and emit-image.ss reuses them.
- Top-level (do ...) is now unrolled form-by-form, like Clojure, so a defmacro
or def in an earlier subform is visible (macro flag set / var interned) before
a later subform is analyzed. This is what makes multi-form -e with a macro work.
Seed is byte-identical (no source references eval), so no re-mint; bootstrap-test
still passes. Zero-Janet corpus 2534 -> 2544 (eval/load-string cases now run),
0 new divergences; floor raised. Prelude corpus, JVM cert, full Janet gate green.
jolt-r8ku
The corpus certifier (test/conformance) flagged four cases where jolt's
hand-written :expected matched a real defect rather than Clojure. Fixed in the
jolt-core overlay, corrected the spec :expected, re-certified against JVM Clojure:
- ex-message: returns nil for a non-throwable (dropped the lenient string branch);
still returns the message for ex-info. (jolt-l8e8)
- munge: preserves the argument's type — a symbol munges to a symbol, not a string.
(jolt-hc35)
- print: (print nil) emits "nil", not "" (top-level nil guard; str yields "").
(jolt-pqio)
- bounded-count: uses the counted? fast path (full count), else counts up to n via
seq — was (min n (count coll)), wrong for counted colls. Added an uncounted-coll
spec case. (jolt-2507)
Removed the 4 :bug entries from known-divergences.edn (now certified), regenerated
corpus + profile, re-minted the Chez bootstrap seed (clojure.core changed). Gates:
Janet 155/0, JVM certify clean, both Chez corpus gates 2534 (floors raised),
bootstrap 6/6, fixpoint intact.
Makes the host-neutral corpus a first-class language specification with
conformance levels, not just a regression suite.
- [suite label] is now a unique, stable case id (extract-corpus disambiguates
duplicate labels with ' (N)' — one collision existed).
- certify.clj --profile emits test/conformance/profile.edn: every non-portable
case classified by the host feature it requires (numerics/double-only,
concurrency/snapshot, host/jvm-interop, host/arrays, host/janet,
async/core-async, runtime/eval, reader/jolt, printer/jolt, strictness/jolt,
impl/representation, bug). 2670 of 2919 cases are portable (pass on any faithful
Clojure); 249 are feature-gated.
- SPEC.md documents the contract: row schema, the JVM oracle, conformance levels,
the feature vocabulary, and a worked new-runtime harness — so hosting jolt
elsewhere and proving it correct is read-one-file mechanical.
Janet gate 155 files 0 failed; certify + zero-janet gates green.
The corpus only ever saw test/spec/*-spec.janet; the 355 hand-written cases in
test/integration/conformance-test.janet (inline Janet, the lazy-seq / IFn /
destructuring / transducer essentials) were invisible to it and to any non-Janet
runtime. extract-corpus.janet now also pulls that (def cases ...) vector, deduped
by :actual, organized into 41 'conformance / <section>' suites recovered from the
file's ### headers. Corpus 2658 -> 2919 rows (+261 unique).
JVM certification: only 1 new divergence ((/ 2) => 0.5 vs 1/2, the all-double
numeric model) — classified. Chez gates: +1 known host gap (instance? Atom, atom
class identity, Phase 4) allowlisted in both runners; parity rose 2295 -> 2533 on
both, floors raised. Janet gate 155 files 0 failed; certifier green (0 new/stale).
Deferred: 41 non-literal core-async spec rows ((a "src") async-harness wrapper)
need harness context the corpus format doesn't carry — left for inc3.
The corpus carried hand-written :expected values — a regression suite but a weak
spec (it checked jolt against its authors, not against Clojure). certify.clj runs
every corpus row's :actual and :expected through reference JVM Clojure 1.12.5 (fresh
user ns per case, output/stdin sunk, 5s per-case watchdog) and compares with =.
Result: of ~2487 vanilla-certifiable rows, 2416 match real Clojure exactly. The 71
divergences are all classified in known-divergences.edn — mostly deliberate
jolt-specific/host-model deltas (all-double numerics, snapshot concurrency, no-JVM
host model, jolt reader features, printer, strictness), plus 4 genuine bugs filed
as beads (jolt-l8e8 ex-message, jolt-hc35 munge, jolt-pqio print-nil,
jolt-2507 bounded-count).
certify-test.janet gates it: skips without clojure on PATH, else fails only on a
NEW (unclassified) divergence or stale allowlist entry; flaky timing-dependent
cases (future-cancel) tolerated either way. Full gate 155 files 0 failed.
The runtime counterpart to bootstrap.ss. host/chez/cli.ss loads the checked-in
seed + the zero-Janet spine and compiles+evals a -e expression entirely on Chez;
bin/joltc execs it. With the seed checked in, a clone runs jolt with only Chez
installed — no Janet at build or run time. Multi-form -e wraps in (do ...) to
match Clojure. test/chez/cli-test.janet 9/9.
Makes the inc8 fixpoint the actual build. host/chez/bootstrap.ss loads a seed
(prelude, image) pair and rebuilds the clojure.core prelude + compiler image from
source via the on-Chez compiler — read/analyze/emit all on Chez, zero Janet.
The seed (host/chez/seed/{prelude,image}.ss) is the checked-in bootstrap
compiler, minted once via the fixpoint (driver/mint-chez-seed* iterates
bootstrap.ss from the Janet-emitted pair to a joint byte-fixpoint). It's a joint
fixpoint: rebuilding from an up-to-date seed reproduces it exactly. So a fresh
checkout + Chez (no Janet) yields a working jolt.
test/chez/bootstrap-test.janet spawns only chez, asserts the rebuilt artifacts
match the seed byte-for-byte and compile+run real cases. Drift (seed sources
changed) fails the test with a re-mint pointer; host/chez/seed/README documents
re-minting.
Extends the fixpoint beyond the compiler image to the whole emitted system.
emit-image.ss now handles macros (defmacro -> bare expander fn + def-var! +
mark-macro!) and re-emits the clojure.core prelude (all tiers + stdlib) on Chez
via jolt-emit-prelude; driver's emit-image-on-chez takes an emit-fn arg.
The prelude converges at pstage3==pstage4 (one stage later than the compiler's
stage2==stage3) because macro expanders bake an auto-gensym id at emit time, so a
Janet-emitted macro carries a different id than a Chez-emitted one — only once
both stages load a Chez-emitted prelude does it stabilize.
fixpoint-test now proves: compiler stage2==stage3, prelude pstage3==pstage4, and
the fully Chez-emitted system (Chez prelude + Chez image, no Janet artifact in the
loop) compiles+runs real cases. 10/10.
The zero-Janet spine proves the on-Chez analyzer/emitter compile arbitrary
Clojure faithfully. This proves the stronger property: the on-Chez compiler
reproduces itself. emit-image.ss re-emits the compiler sources (jolt.ir +
jolt.analyzer + jolt.backend-scheme) ON CHEZ via the loaded image; feeding it
stage1 (the Janet cross-compile) yields stage2, feeding stage2 yields stage3.
stage2 and stage3 are byte-for-byte identical, and stage2 is a working compiler
(real cases compile+run through it). stage1 differs from stage2 only in gensym
numbering, so the fixpoint is stage2==stage3.
driver: emit-image-on-chez / program-emit-image spawn a fresh chez per stage
(clean gensym state). test/chez/fixpoint-test.janet gates it (skips without chez).
(require '[clojure.string :refer [blank?]]) then an unqualified blank? now
resolves. chez-register-spec! registers :refer names (in addition to :as) into a
refer table; hc-resolve-cell's unqualified branch consults it before clojure.core.
Zero-Janet corpus parity 2293 -> 2295 = the Janet-hosted oracle's exact pass
count. The self-hosted Chez compiler (read -> analyze -> emit -> eval, no Janet)
now compiles every corpus case the Janet-hosted compiler does, with 0
divergences. Remaining failures are shared runtime breadth (host interop,
futures, runtime eval) deferred to Phase 4 / jolt-r8ku. Floor 2295.
The batched zero-Janet runner wrote one case per TSV line, so a multi-line
source (e.g. a ;comment\n inside a map literal) split the line and the case was
truncated -> a false "apply non-procedure" crash. Escape \n/\t/\\ when writing
the cases file and unescape in the runner before eval.
Zero-Janet corpus parity 2288 -> 2293 (the 5 comment-in-map cases), 0
divergences — now within 2 of the Janet-hosted oracle (2295). Floor 2293.
Reader gaps the Chez-hosted analyzer hit where the Janet reader didn't:
- ##Inf / ##-Inf / ##NaN symbolic literals (## dispatch -> flonum).
- #(...) anonymous fn shorthand -> (fn* [p__N#] body), with % / %N / %& and the
max-positional arity rule; scans + rewrites list/vector/set/map bodies.
- #?(...) reader conditional: feature set {:jolt :default}, first matching clause
wins. #?@ splicing not yet supported (one niche case allowlisted).
- (ns name (:require [a :as x])) — the require pre-scan now also reads aliases
from an ns form's :require/:use clauses, not just bare (require ...).
Zero-Janet corpus parity 2240 -> 2288, 0 divergences (2 now-reachable cases
allowlisted: str of Infinity inside a collection — same as the prelude gate —
and #?@ splice). spine-test 35/35; prelude parity 2295 unchanged, 0 new
divergences.
(require '[clojure.string :as s]) then s/foo crashed "Unknown class s": the
Chez analyzer resolved a qualified symbol's ns literally, and there was no
alias table (ns.ss jolt-ns-aliases was a stub, require a no-op). Add an alias
table + chez-register-spec! (parses [ns :as a]) in ns.ss; hc-resolve-cell now
resolves a qualified ns through it; compile-eval pre-registers a form's
require/use :as aliases before analysis (analysis precedes the runtime require,
so the runtime require staying a no-op is fine). The batched gate runner clears
the alias table between cases.
Zero-Janet corpus parity 2159 -> 2240, 0 divergences. spine-test 35/35.
Point the Chez-HOSTED analyzer at the full parity corpus (read -> analyze ->
emit -> eval, all on Chez, no Janet) and close the divergences so the
self-hosted compiler is faithful: 0 divergences, 2159/2494 pass.
Keystone: the on-Chez emitter ran with prelude-mode off, so every call to a
non-native clojure.core fn tripped the "unsupported stdlib fn" out-of-subset
guard. The zero-Janet spine always has the full prelude loaded, so turn
prelude mode on in compile-eval.ss (22% -> 84% pass on a sample).
Faithfulness fixes (each was the Chez host/reader diverging from the Janet
analyzer; fixed in the keeper, not the seed):
- emit-const read a char's codepoint via (get v :ch) — the Janet rep; on Chez a
char is native. Route through a new form-char-code host-contract fn (41 cases).
- next over a lazy seq returned the empty-list terminator (truthy), not nil, so
butlast and other (if (next s) ...) loops ran one step too far — broke
some->/some->>/cond->>.
- reader: radix literals (2r1010/16rFF/36rZ), #^ deprecated metadata, ^meta on
collections (lowers to a runtime with-meta form like the Janet reader),
map-literal source order (values eval left-to-right), and nested syntax-quote
over a literal collapses at read time.
- keyword "a/b" splits into ns/name like the seed (destructure {:keys [x/y]}).
- form-syntax-quote-lower implemented on Chez (was a throwing stub).
7 divergences allowlisted: the same print-method-multimethod / host-class set
the prelude gate defers. 328 crashes remain = shared runtime breadth (host
interop, missing core fns, eval/load-string) deferred to Phase 4 / jolt-r8ku,
not compiler faithfulness.
Gate + speedup: test/chez/run-corpus-zero-janet.janet (floor 2159). Its batched
runner (driver/eval-corpus-zero-janet) runs every case in ONE chez process —
load the runtime once, guard + reset the user namespace per case — instead of a
fresh process per case: 1379s -> 1.6s.
spine-test 35/35; Janet gate 151/0; prelude parity 2295/2494 unchanged, 0 new
divergences.
The on-Chez analyzer (inc6a) skipped macros, so let/when/->/defn didn't
expand from source. Each core/stdlib defmacro now emits into the prelude as
(def-var! ns name <expander fn>) + (mark-macro! ns name); form-macro?/
form-expand-1 on Chez look up the macro flag (rt.ss var-macro-table) and
apply the expander to the unevaluated arg forms, and the analyzer re-analyzes
the result. The expander's syntax-quote template was lowered to construction
code at cross-compile time, so it builds the expansion via __sqcat/__sqvec/
__sqmap/__sqset/__sq1 (new host/chez/syntax-quote.ss) as Chez reader forms.
Emit the bare (fn ...), not (def NAME (fn ...)): analyzing a def would
host-intern! NAME as a non-macro stub in the build ctx, and that stub makes a
later (require '[stdlib-ns]) skip loading the real macro — with-pprint-dispatch
then resolved as a fn and returned its unexpanded template. Wrapping the
lambda in def-var! manually never interns NAME. Fuller build-ctx isolation
(so stdlib cases pass instead of crash) tracked in jolt-lpvi.
__sqset builds a real set VALUE, not the reader's tagged-set form — a runtime
`#{~@xs} must be a set, not a map. form-set? additionally recognizes a pset so
a macro template's #{...} expansion still re-analyzes as a set literal.
spine-test 35/35 (20 macro cases: when/when-not/let/->/->>/and/or/cond/if-not/
defn run zero-Janet from source). Prelude parity 2280->2295, 0 new divergences.
Full Janet gate green.
Cross-compile jolt.ir + jolt.analyzer + jolt.backend-scheme to Scheme def-var!
forms via the existing Janet emit pipeline (driver/emit-compiler-image) and run
them ON CHEZ over a Scheme jolt.host impl. A macro-free Clojure expression now
compiles and runs with no Janet in the loop: read (reader.ss) -> analyze
(jolt.analyzer on Chez) -> IR -> emit (jolt.backend-scheme on Chez) -> eval.
host/chez/host-contract.ss is the jolt.host contract on Chez (the portable seam
the cross-compiled analyzer/emitter call): form-* over the Chez reader's forms,
resolve-global/compile-ns/host-intern!/late-bind? over the var-cell registry. ctx
is an opaque record carrying the compile ns. Native-op names are declare-var!'d
into clojure.core so +, *, <, ... classify as :var and the emitter's native-op
path lowers them. form-macro? is a #f stub and macro expansion / syntax-quote /
record hints are stubs for inc6b (runtime macros, jolt-r8ku).
host/chez/compile-eval.ss is the spine entry (read-string -> analyze -> emit ->
eval). driver gains emit-compiler-image / ensure-compiler-image (image caching)
and program-zero-janet / eval-zero-janet.
Two bugs fixed in the keeper, not reproduced:
- the Chez reader stores an unqualified symbol's ns as #f, but the analyzer tests
(nil? ns); hc-sym-ns normalizes #f/'() -> jolt-nil. Without it every handled
special (if/do/fn*) misanalyzed as a plain invoke.
- char (int->char) was missing from clojure.core on Chez; the emitter's
chez-str-lit needs it for keyword/string consts. Added jolt-char to converters.ss.
Gate: test/chez/spine-test.janet 15/15 (Chez-hosted analyzer value == Janet-hosted
oracle through the same emitter/RT; only the analysis host differs). Full Janet
gate green (150 files). driver.janet is in the prelude fingerprint so the cache key
moved; prelude content is unchanged. jolt-chez fingerprint/ensure-prelude made
public for the test harness.
read-all reads every top-level form of a string (the parse-all the compile path
needs in inc 6). With this the portable reader covers everything reader.janet does
(atoms 5a, collections+quote/meta 5b, dispatch 5c, multi-form 5d).
Validated: reader-parity 149/149 (every construct); jolt.reader compiles and runs
on build/jolt ((require [jolt.reader]) (read-all …) works natively). The
interpreted reader overflows the eval stack on large/deeply-nested files, so real-
file/full-corpus validation happens when it's cross-compiled and run on Chez
(inc 7). Position tracking (checker) and the actual wire-in replacing reader.ss are
deferred — they ride inc 6 (jolt.reader on Chez). See new beads.
Ports the full # dispatch to the portable reader: #{} sets, #() anon-fns, #?/#?@
reader-conditionals, #_ discard, #' var-quote, #"" regex, #inst/#uuid/#tag tagged
literals, ## symbolic (Inf/-Inf/NaN), and #^ deprecated metadata. With this the
reader is feature-complete except position tracking + wire-in (inc 5d).
Reader-conditionals resolve clause-order against a portable feature set (atom
#{:jolt :default}); #? -> :skip / :form, #?@ -> :splice (the control protocol from
5b). #() uses the two-pass %-scan (collect indices, then rebuild replacing %N/%/%&
with gensym params) over the form tree via the jolt.host form-* contract. Three
host constructors added: form-make-set, form-make-tagged, form-gensym-name.
reader-parity 149/149. #() compares modulo gensym (canonicalize #-suffixed param
names by first-occurrence order — the two readers gensym different names but the
structure + %-mapping must match). ##NaN checked by the NaN!=NaN property. Full jpm
gate green (prelude pre-warmed). jolt-9ufe.
Ports list/vector/map literals and the quote family (' ` ~ ~@ @) + metadata (^)
to the portable Clojure reader. read-form now returns a [kind payload pos] control
triple (:form / :skip / :splice) instead of the Janet reader's :jolt/skip sentinel
FORMS — out-of-band control is collision-free and host-neutral (no tagged struct
to build or recognize). read-delimited dispatches the kinds; read-next-form skips
comments where a single datum is needed; read-map pairs k/v skipping trivia in
either slot. syntax-quote of a self-evaluating literal collapses at read time.
Four host constructors added to the contract (host_iface): form-make-list/vector/
map + form-sym-merge-meta (attach ^meta to a symbol). form-make-map reuses the
seed's reader-map (now public) for the source-order kv tracking. The portable
reader accumulates items in a jolt vector and the host builds its native form rep.
Gate: reader-parity 107/107 (lists/vectors/maps incl. nested + comments-in-coll,
quote/syntax-quote-collapse/unquote/deref, ^:dynamic/^Type/^{} meta). Full jpm gate
green (prelude cache pre-warmed — a cold cache races under the parallel gate when
the jolt-chez fingerprint changes; pre-existing, see new bead). jolt-sh1n.
fix-bugs-dont-reproduce, scoped per the keeper rule: jolt-if19 (a leading + on a
numeric literal errored instead of reading as the positive number) is fixed in
jolt.reader (read-number* now strips a leading + like -, positive), the code we
keep. The Janet seed reader (reader.janet) is left untouched — it's deleted in
Phase 5, so fixing it is wasted work.
Since the seed reader stays buggy, reader-parity can't use it as the oracle for
these inputs: added check-correct to assert the portable reader against the hand-
verified value (+5 => 5, +42, +0xff => 255, +3.5). reader-parity 67/67. No Janet
binary/gate impact (jolt.reader is not yet in the binary path). jolt-if19.
Starts taking the reader off Janet (src/jolt/reader.janet, 831 lines) into
portable jolt-core Clojure. jolt.reader holds the lexing/parsing LOGIC; form
construction + string->number parsing delegate to the jolt.host contract — a
Clojure source file can't write a {:jolt/type :symbol} literal (parses as a
tagged form) and the concrete representation is the host's to own. Same split the
analyzer/emitter already use. Once cross-compiled this runs on Chez so compile-
from-source needs no Janet reader.
inc 5a = the atom layer: whitespace/comments, symbols (+ nil/true/false),
keywords, strings (escapes), numbers (sign/hex/radix/ratio/fractional/exponent,
trailing N/M), characters. Collections, quote/deref/meta and dispatch (#) follow
in 5b/5c (throw not-yet-ported). Positions are char indices (Janet uses bytes);
identical for ASCII and the gate compares form VALUES, not positions.
host_iface.janet gains four reader primitives on the contract: form-make-symbol,
form-make-char, form-char-from-name, form-scan-number (the irreducible host bits
the portable reader rests on). Additive — new jolt.host interns, nothing else
changed.
Surfaced jolt-if19 (Janet seed reader: +N literals error instead of reading as N;
read-number strips only the - sign). The port reproduces it; both-throw counts as
faithful parity in the gate.
Gate: reader-parity 64/64 (symbols/keywords/strings/ints/hex/radix/ratio/floats/
exponent/N-M/chars). Full jpm gate green after clean rebuild, conformance 355x3.
jolt-50xx.
driver.janet now compiles IR via the portable Clojure emitter (jolt.backend-
scheme) instead of emit.janet, at every entry point (compile-program, emit-core-
prelude, eval-e-with-prelude). The emitter is loaded into the ctx and called like
the analyzer. emit.janet stays only as the emit/program string-wrapper until
program assembly ports to Clojure with compile-from-source; its emit fn is no
longer called anywhere (emit-test's truthy-elision helper now uses the new
d/scheme-emit too). This takes the IR->Scheme emitter off Janet.
A form-by-form diff of the two emitters over the whole prelude found one gap:
emit-const missed char literals because a :jolt/type-tagged struct is not a plain
jolt map? — switched to the form-char? host contract. Diff then 0.
jolt-chez prelude fingerprint now includes backend_scheme.clj + host_iface.janet.
Gate: full prelude corpus 2280/2494, NEW divergence 0, same buckets as the Phase-2
emit.janet floor (36 emit-fail, 170 crash) — the Clojure emitter is byte-for-
behavior identical. emit-test 331/331 (now via the Clojure emitter), emit-parity
58/58. jolt-duot.
Completes the op coverage of the portable Clojure emitter — it now handles every
op emit.janet does (const/local/var/the-var/host/host-static/host-new/if/do/
invoke/vector/set/map/quote/throw/try/regex/inst/uuid/host-call/let/loop/recur/
fn/def). Adds emit-try (guard + dynamic-wind), :throw, :regex/:inst/:uuid, and
:host-call (jolt-host-call for rt-shimmed methods else record-method-dispatch).
def-meta + quoted-symbol-meta needed emit-quoted to reconstruct plain jolt VALUES
(metadata maps), not just reader forms. The blocker was that :meta arrived as a
raw Janet table embedded in the IR — jolt's count/map?/keys don't work on a table
(counter to jolt.ir's 'no host values embedded'). Fixed at the host seam:
h-sym-meta now returns the meta as an immutable struct, which is a portable jolt
map (jolt count/map?/keys work on a struct, and the Janet backend's merge/get
still do too). emit-quoted handles both reader forms (jolt.host form-* contract)
and jolt-value collections (native map?/vector?/set?/seq? branches, after the
form-* branches so reader forms win).
Gate: emit-parity 55/55 (incl try/catch/finally, ^:private def-var-with-meta!
structural check, inst/uuid eq, regex smoke, quoted-sym-meta). Full jpm gate
green after clean rebuild (seed change). jolt-me6m.
Adds :vector/:map/:set (emit-ordered, left-to-right element eval) and :quote to
the portable Clojure emitter. Collection-literal nodes carry already-analyzed IR
items so they just recurse; quote walks the RAW reader form.
emit-quoted walks the reader form via the jolt.host form-* contract (form-list?/
form-elements/form-vec-items/form-map-pairs/form-set-items/form-sym-*), the same
portable seam the analyzer uses — not host-native predicates, so it works
unchanged whether the form came from the Janet reader or the Chez reader. Reader
forms are raw host representations (Janet list=array, vec=tuple), so native
list?/vector? would not see them; the contract is the correct abstraction and
keeps the emitter host-neutral. Quoted-symbol metadata and def-meta still defer
to inc 3.
Surfaced a latent Janet-host bug (jolt-tg9s): (quote #{...}) evaluates to the raw
reader form instead of a reconstructed set, so (contains? (quote #{:p :q}) :p) is
false on build/jolt. The Chez emitter is correct (real Clojure: true); the parity
test asserts the verified value for those two cases.
Gate: emit-parity 42/42 (incl vector/map/set literals, coll/kw-as-fn, quoted
list/vec/map/set/symbol/nested). emit-test 331/331, conformance 355x3. jolt-7jvp.
First spine increment of self-hosting the compiler on Chez. The IR->Scheme
emitter is host/chez/emit.janet (Janet); to get the analyzer emitting its own
code on Chez with no Janet, the emitter logic has to be portable Clojure that
cross-compiles and runs on Chez itself.
jolt-core/jolt/backend-scheme.clj ports the core ops: const/local/var/the-var/
if/do/let/loop/recur/invoke (+ native-ops)/fn/def, plus the chez-str-lit/flonum/
munge/truthy-elision helpers and prelude-mode. Output is Scheme source text, op-
for-op with emit.janet. recur-target/known-procs are dynamic vars (auto-restore,
no throw-leak). Quote, collection literals, try/throw, host interop, regex/inst/
uuid and program assembly come in later increments (they throw not-yet-ported).
Gate: test/chez/emit-parity.janet loads the Clojure emitter interpreted on the
Janet host and runs each case through it -> Chez -> compares to the Janet CLI
oracle. 18/18 incl fib, factorial loop, multi-arity, variadic, higher-order,
#() shorthand, the mandelbrot kernel. emit-test 331/331 (emit.janet path
untouched), conformance 355x3. jolt-hg7z.
Close the remaining Phase-2 stdlib parity gaps.
clojure.set: pure Clojure over core, so just added to the prelude stdlib tier
(driver.janet stdlib-ns-files + jolt-chez fingerprint), same as clojure.edn.
clojure.math: not a .clj on the seed (native math/ bindings via jolt-h79), so
Chez gets its own host/chez/math.ss def-var!ing each fn over Chez native flonum
math. The analyzer already knows the ns (api.janet install-clojure-math!), so
refs lower to var-deref. Also adds the missing 'long' coercion to converters.ss
(int's sibling; several math cases wrap in long).
clojure.pprint: dropped the 2-arity's (binding [*out* writer] ...). *out* isn't
a bindable var in jolt — printing routes through the host (dyn :out) seam, so
the binding never redirected anything; it only made the defn uncompilable, which
the seed tolerated via interpreter fallback. Chez has no fallback, so the whole
pprint defn died. Dropping it is behavior-preserving (writer was always ignored)
and lets pprint compile cleanly. Both corpus cases pass.
Corpus parity 2259 -> 2280, crashes 191 -> 170, 0 new divergences. Floor raised.
New unit test test/chez/_stdlib.janet (27/27). Full Janet gate green (147 files).
Chez-side recursive-descent Clojure reader (host/chez/reader.ss) producing the
same jolt forms the Janet reader yields, behind the read-string / __parse-next /
__read-tagged seams the seed registers in eval_runtime.janet. That lights up the
whole *in* read family — read, read+string, with-in-str (read) — plus read-string
and read-string metadata, none of which needed an analyzer change (read-string is
a clojure.core seam, jolt-nil on the prelude until now).
Reader output is pinned to the Janet reader's shapes: numbers coerce to flonum
(the all-double model emit-const uses, else a read int isn't = a source int),
sets read as the {:jolt/type :jolt/set :value [...]} FORM, #tag/#inst/#uuid/#regex
as tagged forms (no data reader applied — read-string never evaluates), ^meta on
a symbol, and the ' ` ~ ~@ @ reader macros. clojure.edn is added to the prelude
tier; its edn->value builds the real set/tagged values and __read-tagged reuses
the inc X #inst/#uuid constructors. The reader-arity edn/read stays a lazy gap
(drain-reader is Janet-coupled) — read-string is the live path.
eval / load-string / runtime defmacro are still out: they need the compiler at
runtime, which is Phase 3 (self-host). Chez-only change, no Janet gate.
Parity 2238 -> 2259, 0 new divergences. _reader 47/47; all chez unit tests green;
emit-test 331/331.
The analyzer lowers a #inst/#uuid tagged form to a :inst/:uuid IR leaf, mirroring
the existing :regex node: the Janet back end punts to the interpreter (its
data-readers parse the literal, so seed behavior is unchanged), the Chez back end
emits jolt-inst-from-string / jolt-uuid-from-string.
host/chez/inst-time.ss is the Chez-native value layer: a jinst record holding
epoch ms (RFC3339 parsed via Hinnant civil/days math, with Clojure's partial
defaults and +/-hh:mm offsets), wired into jolt-get (so the overlay inst?/inst-ms
read it), jolt= / jolt-hash (instant identity as a map key), pr-str (#inst
"...-00:00"), str, type, and instance? java.util.Date. The java.time surface
(DateTimeFormatter ofPattern/ISO_LOCAL_DATE_TIME/ofLocalized*, the pattern engine,
Instant, ZoneId, LocalDateTime, FormatStyle, Locale, Date) ports java_base.janet
over host-static.ss's registries.
Corpus 2202->2238, 0 new divergences; clears the whole 'unsupported form'
emit-fail bucket. Full Janet gate green (analyzer/backend changes are
behaviour-preserving — #inst still parses through the interpreter's data-readers
on the seed).
clojure.java.io/reader as an in-memory StringReader over slurp/string/char[]/
File; File .toURL/.toURI returning a url jhost (.toString/.getPath); slurp drains
a StringReader; char-array; with-open's __close seam over jhost readers and plain
:close maps. All in host/chez/io.ss (Chez-native, no analyzer change). Corpus
2191->2202, 0 new divergences. clojure.edn/read over a PushbackReader stays
jolt-r8ku (runtime read).
A File is a path-backed jfile record: (instance? java.io.File f) is true,
str/slurp coerce it to its path, and the File method surface (getName/
getPath/exists/isDirectory/isFile/listFiles/getParent) dispatches through
record-method-dispatch. slurp/spit/flush run over Chez's filesystem
primitives; file-seq's dir primitives (__file?/__dir?/__list-dir) and the
overlay's File branch (.isDirectory/.listFiles, which emit to jolt-host-call)
are jfile-aware. clojure.java.io/file + as-file are def-var!'d natively.
New host/chez/io.ss, a Chez-native implementation -- the seed's
clojure.java.io (io.clj) is a Janet-backed shim over janet.*/janet.file, so
it can't be reused. The analyzer resolves io/file because the seed ctx has
clojure.java.io loaded; only a runtime def-var! is needed. type/instance-
check/str-render/jolt-host-call are extended via the set!-wrap pattern (type
also re-def-var!'d since the var cell captured the old value).
Reader/StringReader-coupled io (io/reader, line-seq over a file, .toURL,
slurp over a reader) deferred to jolt-at0a.
Parity 2176 -> 2191, 0 new divergences. New test/chez/_io.janet 20/20.
list? was nil on Chez because one cseq record backs both lists and lazy/
realized seqs. Add a list? marker field (cseq v2) set only on the HEAD cell
of a list -- (list ...), quoted list literals, cons, reverse, conj onto a
list. rest/next/seq/map therefore yield unmarked seq cells, so they are
seqs and not list?, matching the seed (where rest-of-a-list is a non-list
seq). Empty () is treated as a list.
vector?: drop the map-entry exclusion. Clojure's MapEntry implements
IPersistentVector and the seed agrees -- (vector? (first {:a 1})) is true.
Only dot-forms' coll dispatch read jolt-vector?, where a 2-vector entry is
correct.
clojure.walk + clojure.template join the prelude stdlib tier. The driver
now evals each stdlib ns's requires -- and the ns form's (:require ...)
clause -- so an aliased ref (template's walk/postwalk-replace) resolves at
emit time instead of lowering to an Unknown class host-static. ns forms are
evaled for that side effect but not emitted, so the runtime *ns* doesn't
leak to the last stdlib ns.
Parity 2163 -> 2176, 0 new divergences. New test/chez/_walk.janet 39/39.
Bare class names (String, Keyword, File...) evaluate to their JVM
canonical-name string, the same value (class x) returns, so
(= String (class "x")) holds and (defmethod m String ...) keys match a
(class ...) dispatch. New host/chez/host-class.ss ports
eval_resolve.janet's class-canonical-names + core_refs.janet's core-class
(scalar arms; collections/seqs are host-taxonomy-dependent and not class-
compared in the corpus).
The analyzer already resolves these names to clojure.core vars (the seed
ctx interns them via setup-class-ctors), so the back end emits
(var-deref "clojure.core" "String") and a runtime def-var! is all that's
needed -- no analyzer change, Janet path untouched. The class native MUST
land together with token resolution: alone it turns the bare-token corpus
cases (562/564) into divergences (this bit last session).
Parity 2154 -> 2163 (cases 560/562/563/564/2500-2503), 0 new divergences.
New test/chez/_class.janet 19/19.
The Chez atom record gains watches (an alist of key->fn) and validator
slots. swap!/reset! now validate the candidate value before storing and
notify watches after, in the seed's order (core_refs.janet) — the watch fn
is called (key ref old new). compare-and-set!/swap-vals!/reset-vals! route
through reset!/swap! so they validate + notify too.
add-watch/remove-watch/set-validator!/get-validator are native here and
re-asserted in post-prelude.ss: the clojure.core overlay implements them via
jolt.host/ref-put! on (get atom :watches), a Janet-table mutation a Chez atom
record can't answer, so its def-var! would otherwise clobber these. set-
validator! validates the current value immediately (Clojure throws if already
invalid).
Parity 2150 -> 2154, 0 new divergences. New test/chez/_atomwatch.janet 10/10.
The (class x) native and the clojure.walk port were explored this increment
but deferred: class alone makes the String-class-token corpus cases emit-and-
run with a wrong value (the bare token doesn't resolve yet) — filed jolt-13zk
to land the native together with token resolution; clojure.walk is blocked on
list? + map-entry-as-vector (jolt-75sv).
The analyzer lowers the `.` special form (. target member arg*) and the
.-field field-access head to a :host-call instead of leaving them
uncompilable. Janet behaviour is unchanged — its back end punts :host-call
to the interpreter, which re-runs the original `.` form via eval-dot.
The Chez back end routes a non-shimmed :host-call through
record-method-dispatch, extended by a new host/chez/dot-forms.ss with the
arms dispatch-member covers but the record/string base did not, mirroring
src/jolt/interop/collections.janet precedence:
- collection interop first (count/seq/nth/get/valAt/containsKey on a
vector/map/set), so (. {:count 9} count) is the entry count like the seed
- field access for a "-name" member (records and maps)
- the seed's universal object-methods (getMessage/getCause/toString/
hashCode/equals) on a non-record map, winning over a field lookup
- non-record map member: a stored fn is a method called with self, else
the field value
Raw seqs are excluded from coll interop — the seed's behaviour there is
representation-dependent (plain (seq v) vs a lazy-seq) and a normalized cseq
can't mirror it. Also added getMessage/getLocalizedMessage/equals to the
string method surface so a thrown string / Exception. ctor (which keeps the
message string) answers .getMessage.
Parity 2134 -> 2150, 0 new divergences. New test/chez/_dotform.janet 26/26;
emit-test 331/331.
Lower host class interop on the Chez back end. The analyzer now turns a
non-var qualified ref `Class/member` into a :host-static node and a
`(Class. ...)` / `(new Class ...)` form into a :host-new node (ir.clj
gains both, with walker support). The Janet back end punts both to the
interpreter, so its behavior is unchanged (verified: dot-form, `..`
threading, shadowed `new`, and all interop still resolve via fallback).
The Chez emit lowers a value ref to host-static-ref, a call head to
host-static-call, and a constructor to host-new. host/chez/host-static.ss
is the runtime registry these resolve against — the Chez port of the
seed's class-statics / class-ctors / tagged-methods (java_base.janet +
host_io.janet), restricted to the java.lang/util/net/io surface portable
cljc code calls: Math, System (getenv/getProperty/exit/currentTimeMillis),
Long, Integer, Boolean, Character, String, Thread, Class, Pattern
(compile/quote/MULTILINE), URLEncoder/Decoder, Base64, the Number method
surface (byteValue/intValue/...), plus the StringBuilder, StringWriter,
StringReader, PushbackReader, HashMap, StringTokenizer, BigInteger,
String, MapEntry, and exception constructors. Constructed objects are
jhost records dispatched through record-method-dispatch.
Also: emit now evaluates collection-literal elements left-to-right
(emit-ordered) — Chez evaluates call args right-to-left, which had been
swapping side-effecting elements in [(read r) (read r)] and map literals.
This un-allowlisted the 6 eval-order corpus cases (the read-line trio +
the three map-construction cases). Removed `.write` from the
jolt-host-call fast-path so a StringWriter routes through dispatch.
java.time formatting, edn/read-over-readers, and slurp/with-open over
readers are deferred to a follow-up.
Corpus parity 2078 -> 2134 (floor raised), 0 new divergences; the
print-method builtin-override case is allowlisted (same multimethod gap,
newly reachable now that StringWriter constructs). emit-test 326/326,
_javastatic 51/51, conformance 355x3, full jpm test green.
Bring the clojure.string namespace up on Chez so aliased refs like s/split,
s/upper-case, s/join, s/replace resolve and run.
Three pieces. (1) The Chez AOT driver analyzes the whole user form before any
require runs, so a (require '[clojure.string :as s]) never registered the
alias in time; eval-e-with-prelude now recursively pre-evals require/use forms
against the ctx, which loads the aliased ns and registers the alias so the
analyzer resolves s/X to a clojure.string var. (2) emit-core-prelude emits
stdlib namespaces (clojure.string) as their own def-var! tier through the same
analyze->emit pipeline, so the runtime var-deref resolves. (3) natives-str.ss
def-var!s the str-* primitives clojure.string.clj is written over (upper/lower/
trim/triml/trimr/find/reverse-b/join/split/replace/replace-all), plus no-op
require/use. Regex split keeps interior empties and honors the limit (ported
the seed re-split); regex replace does $N backref expansion and fn replacement
(ported replacement-for). new RT/clj files added to the prelude fingerprint.
Corpus prelude floor 2026 -> 2078 (+52), 0 new divergences. _strns 28/28 vs
build/jolt. Four previously-CRASHING cases now emit+run and surface pre-existing
gaps (read-line vector eval-order x3, instance? clojure.lang.Atom) — allowlisted
with notes. full jpm test + conformance x3 green.
Port the java.lang.String/CharSequence method surface to the Chez RT so
(.toUpperCase s), (.substring s a b), (.indexOf s x), the regex methods
(.matches/.replaceAll/.replaceFirst/.split), etc. run on a string target.
natives-str.ss holds jolt-string-method, ported from the seed's surface in
eval_resolve.janet: ASCII case mapping (byte-oriented like the seed), -1 on
indexOf miss, flonum numeric returns to match jolt's number model, Scheme
chars for charAt, and the regex methods over the irregex compiled via
jolt-re-pattern. record-method-dispatch gains a string? arm that falls
through to it (unsupported methods still throw).
Corpus prelude floor 2002 -> 2026 (+24), 0 new divergences. _str 27/27 vs
build/jolt; full jpm test + conformance x3 green.
The (. x m) dot-form (the . special form, distinct from .method sugar) and
the clojure.string namespace (needs prelude plumbing + Pattern) stay deferred.
Implement (type x) on the Chez RT (host/chez/natives-meta.ss). Mirrors the
seed's core-type: the :type metadata wins when present, a record yields its
ns-qualified class-name SYMBOL (user.TyR — no-ns sentinel #f so it = the
overlay's (symbol (str t))), everything else a host-taxonomy keyword
(:number/:string/:vector/:map/:set/:seq/:fn/…). Total by construction — a
non-record value falling through to a crash would read as a divergence, so the
cond covers every value type incl. the host wrappers (atom/volatile/regex/var/
transient/uuid -> :jolt/*, a :jolt/type-tagged map like ex-info -> its tag,
sorted-set -> :jolt/sorted-set, sorted-map -> :map) and a final :object.
Also pin sequential?/seq? on lazy seqs (test/chez/_seqpred.janet): the inc M
seq? re-def-var! fix already covers sequential? transitively (sequential? is
overlay and delegates to seq?), so no code change — the earlier "still broken"
note was wrong, it assumed sequential? was native like seq?.
Prelude corpus parity 2000 -> 2002 (the two type cases), floor raised, 0 new
divergences. Gate: _type 37/37 + _seqpred 22/22 (both vs build/jolt oracle),
emit-test 321/321, full jpm test, conformance 355x3.
Add the dynamic-binding cluster on the Chez RT: a per-thread binding stack
(host/chez/dyn-binding.ss) backing binding / with-bindings* / var-set /
thread-bound? / with-local-vars / with-redefs / bound-fn* /
get-thread-bindings / alter-var-root + the __local-var seam.
Frames are stored innermost-first as identity-keyed alists of mutable
(cell . value) pairs, so var-set updates the current binding in place. The
two var-read paths — var-deref (compiled code) and jolt-var-get (var-get /
deref on a cell) — are chained onto the stack so a `binding` frame is seen
by every read, with a fast path when the stack is empty. var-cells now hash
by ns/name so a var works as a map key (with-redefs builds (hash-map (var f)
v); get-thread-bindings returns a var-keyed map).
Fix a latent bug exposed once with-in-str could bind *in*: seq? didn't
recognize a lazy-seq. predicates.ss's jolt-seq? predates the lazyseq record,
and unlike the native-op dispatchers it's reached via var-deref, so the
patch must re-def-var! the var, not just set! the top-level binding.
Note: with-bindings* over a hash-map literal now returns the correct Clojure
value where the seed returns a stale one — the seed's PHM can't find a var
key (which is why its `binding` uses array-map); on Chez frames look up by
cell identity.
Prelude corpus parity 1972 -> 2000, floor raised. Gate: _dynbind 24/24,
prelude corpus 0 divergences, full jpm test, conformance 355x3.
Capture a def's reader metadata on the Chez var. The :def emit now lowers a def
with non-empty metadata to def-var-with-meta!, which stores the user meta
(^:private / ^Type tag / docstring -> {:doc}) in an eq side-table keyed by the
var-cell. jolt-meta of a var-cell merges that onto {:ns :name} derived from the
cell, so every var reports {:ns :name} like Clojure with the def-time meta
layered on. (^{:map} metadata on a def name stays uncompilable for the compiler
generally — analyzer rejects it, the Janet back end punts to its interpreter,
which Chez lacks — so it's out of subset, not a meta-capture gap.)
Added natives-meta.ss to the prelude-cache fingerprint. Prelude parity
1969 -> 1972, 0 new divergences; the three var-metadata allowlist entries
(^:private / ^Type tag / docstring) dropped. New focused gate
test/chez/_var_meta.janet.
Reimplement the ctx-coupled seed ns natives over the rt.ss var-table, since
Chez has no ctx. host/chez/ns.ss adds a jns namespace value + a registry and
binds find-ns/the-ns/create-ns/in-ns/all-ns/ns-publics/ns-map/ns-interns/
ns-aliases/resolve/find-var/ns-unmap/*ns* into clojure.core.
The resolve friction: native-ops (+, map, …) are inlined at emit so they have
no var-cell, and (resolve '+) was nil — diverging from Clojure where it's a
var. Added a defined? flag to the var-cell record (set by def-var!/declare-var!,
left false on a lazily-materialised forward ref) and def-var!'d every native-op
name to its value-position proc, so resolve returns the cell iff genuinely
defined. ns-unmap clears the flag. resolve never interns an empty cell
(var-cell-lookup is non-creating).
ns-name is overridden natively in post-prelude (the overlay reads
(get ns :name), nil on a jns record); the printers render a namespace as its
name. *ns* binds to the user ns; in-ns re-binds it. use/require cross-ns
switching stays deferred to Phase 3 (the analyzer bakes a def's target ns at
compile time).
Prelude parity 1951 -> 1969, 0 new divergences; four now-passing allowlist
entries dropped (ns *ns* cases + str-of-a-var). New focused gate test/chez/
_ns.janet (19 cases, expectations from the JVM-canonical build/jolt).
inc I (jolt-n7rz) — vars as first-class objects. (var x)/#'x lowered a
:the-var IR op the Chez emitter didn't handle (57 emit-fails, the biggest
bucket); emit it to the rt.ss var-cell and shim the static var ops in
vars.ss: var?/var-get/deref/var-as-IFn/var =/pr-str (#'ns/name) + a native
bound? (the overlay reads (get v :root), nil on a var-cell record). def now
RETURNS the var (#'ns/name) like Clojure — def-var!/declare-var! yield the
cell, not the value — so (var? (def x 1)) is true and pr-str-of-var/defn
pass (un-allowlisted). Dynamic binding (binding/with-bindings*/var-set/
thread-bound?) stays a follow-up; those crash on nil host prims (safe).
Var def-time metadata (^:private/^Type/doc) isn't captured yet — allowlisted.
inc J (jolt-snry) — scalar natives. natives-misc.ss: a juuid record
(random-uuid v4 / parse-uuid / uuid? / #uuid pr-str / str), a %-format
engine (%d/%s/%.Nf/%x/%c/%%; printf rides on it), a jtagged record
(tagged-literal + :tag/:form + #tag pr-str), bigint/biginteger as
near-identity over the all-flonum model. Overlay names (uuid?/random-uuid/
parse-uuid/tagged-literal?) re-asserted in post-prelude.ss.
Prelude parity 1898 -> 1951, 0 new divergences. Floor raised to 1951.
sequence and transduce were seed natives nil on the prelude; the stateful
transducer arities (take-nth/map-indexed/partition-by, overlay) drive a
volatile via volatile!/vswap!/vreset!/deref, also unshimmed — so the whole
(sequence xform coll) / (transduce xform f coll) surface crashed.
natives-xform.ss: native volatiles (a jvol record + volatile!/vreset!/
vswap!/volatile? + a deref arm); transduce/sequence built on the existing
into-xform / reduce-seq. The overlay vreset!/vswap!/volatile? drive a
volatile through ref-put!/:jolt/type (tagged-table only), so they're
overridden natively in post-prelude.ss.
driver.janet: drain each chez pipe to EOF instead of a single ev/read. A
program with a stdout side effect (newline) flushes in two writes and a
lone ev/read sometimes caught only the first chunk, dropping the trailing
value — an intermittent gate divergence. Now reads until the pipe closes.
Prelude parity 1886 -> 1898, 0 new divergences. Floor raised to 1898.
The lazy-seq macro expands to (make-lazy-seq (fn* [] (coll->cells body)))
and lazy-cat to (concat (lazy-seq c) ...); both seed natives were nil on
the prelude, so every overlay fn built on lazy-seq — repeat/iterate/
cycle/dedupe/take-nth/keep/interpose/reductions/map-indexed/distinct/
interleave/tree-seq(->flatten)/partition-all/lazy-cat — hit apply-jolt-nil.
lazy-bridge.ss bridges to the cseq model: a jolt-lazyseq is a deferred
seq forced once by an extended jolt-seq; jolt-cons defers a lazyseq tail
so an infinite (repeat/iterate/cycle) stays lazy. A lazyseq is a new
value type, so the dispatchers that don't route through jolt-seq learn it
(sequential? for =/hash, plus count/empty?/nth/printers) or a raw
unrealized lazyseq escapes — the corpus compares (= [1 3 5] (take-nth …))
against it directly.
seq.ss: jolt-concat is now fully lazy (the rest isn't forced until the
first coll is exhausted), so a self-referential lazy-cat — fib =
(lazy-cat [0 1] (map + (rest fib) fib)) — no longer memoizes its tail as
empty by reading fib before its def binds.
Prelude parity 1837 -> 1886, 0 new divergences. Floor raised to 1886.
jolt.host/tagged-table, ref-put!, ref-get resolved to jolt-nil on the
prelude, so the 25-sorted tier and every overlay fn that calls sorted?
(empty, ifn?, reversible?, map?, set?, coll?) hit apply-jolt-nil.
host-table.ss provides the three primitives over a Chez mutable htable
record and set!-extends the collection dispatchers (seq/count/get/
contains?/assoc1/dissoc/conj1/disj/empty?/keys/vals/coll?/map?/set? +
printers + equality + value-host-tags) with a sorted arm routing through
each value's :ops table — the seed dispatch pattern, same as records.ss.
first/rest/next fall out free since they seq first. Sorted colls print
in sorted order ({k v, k v} for maps) and = canonicalize like their
unordered counterparts.
emit.janet: a computed call operator (an :invoke/:if expr that yields an
IFn, e.g. ((sorted-map :a 1) :k)) now routes through jolt-invoke instead
of a raw Scheme application.
Prelude parity 1723 -> 1837, 0 new divergences. Floor raised to 1837.
meta/with-meta resolved to jolt-nil. Chez values don't carry metadata, so
collections use an identity-keyed side-table: with-meta returns a fresh copy of
the value (new identity) and records its meta there, leaving the original
unchanged (immutable-with-meta) and dropping meta on a later copying op. Symbols
carry meta in their existing field; meta on a non-metadatable value is nil.
vary-meta works over these. with-meta on a fn stays fn? (jolt is lenient).
emit.janet carries a quoted symbol's reader metadata (^:foo bar) onto the
emitted jolt-symbol so (meta 'x) sees it; symbol = still ignores meta.
Prelude parity 1701 -> 1723, 0 new divergences. jolt-rkbc.
The largest remaining crash bucket: defrecord/deftype/defprotocol/extend-type/
reify. make-deftype-ctor/make-protocol/protocol-dispatch/register-method/
satisfies?/extenders/instance-check/make-reified were ctx-capturing seed natives
resolving to jolt-nil.
records.ss adds a jrec type (tag + field alist), set!-extended into every
collection dispatcher (get/=/hash/count/keys/vals/seq/assoc/conj/contains?/map?/
pr-str/pr-readable/str) via the transients.ss capture-prev pattern, plus a
protocol registry (type-tag -> proto -> method -> fn) and dispatch over record
tags / host-type candidates. (get rec :jolt/deftype) returns the tag, so the
overlay record? works unchanged. A record equals another of the same type with
equal fields, is map?/coll? not vector?, prints #ns.Name{...}, and str uses a
custom Object toString impl when defined.
emit.janet :host-call now routes a non-shimmed method to record-method-dispatch
(was an emit-fail), so (.protoMethod record args) compiles; .-field access is
still analyzer-punted (deferred).
Prelude parity 1652 -> 1701, 0 new divergences. 4 print-method-multimethod cases
moved crash->allowlist (the printer doesn't consult a custom print-method yet).
jolt-jgoc.
__pr-str1/__write/__with-out-str/__eprint/__eprintf resolved to jolt-nil, so the
whole overlay print family (pr-str/pr/prn/print/println/print-str/prn-str/
println-str) hit the apply-jolt-nil crash bucket. jolt-pr-str is str-style
(strings raw); pr-str needs readable (pr) style — strings quoted+escaped at every
level. printing.ss adds the readable renderer (mirrors jolt-pr-str, recurses,
delegates scalars), plus the infinities long-form and a transient arm (those
were crash->divergence reveals once pr-str ran).
jolt-9zhh.
set/hash-map/hash-set/array-map/rand resolved to jolt-nil on the prelude
(the apply-jolt-nil crash bucket) — the pmap/pset ctors already existed in
collections.ss, just bind the public clojure.core names to them.
Map entries are now a distinct type: a pvec carries an ent flag (default #f),
so an entry equals its [k v] vector and walks like one (nth/count/seq/=/hash/
print read only v) but is not vector? and is map-entry? — matching Clojure's
MapEntry. seqing a map produces flagged entries; vector? excludes them. This
unblocks key/val (overlay fns gated on map-entry?) and the every? map-entry?
cases.
Prelude parity 1534 -> 1593, 0 new divergences. jolt-agw6.
Elide the redundant jolt-truthy? wrapper on an :if test that provably
yields a Scheme boolean (a native comparison/not, or a boolean const).
Sound because jolt-truthy? of #t/#f is identity. The hot fib/mandelbrot
tests are all comparisons, so this is a direct ceiling lever: fib(30)
end-to-end 24.0 -> 14.4 ms.
Add bench-pipeline.janet (JOLT_CHEZ_BENCH=1, opt-in) timing fib(30) +
mandelbrot(200) through the real pipeline vs the spike ceiling.
Mandelbrot 200 runs at 87 ms, at/below the 98 ms generic-flonum ceiling
- the substrate ceiling is reached end-to-end. fib sits at 2x its
hand-flonum ceiling; the residual is jolt's all-double number model
(typed fl*/fx* emission is Phase 4). Compile-only is total for the
compute subset (every form emits; Chez has no interpreter fallback).
Full parity unchanged at 1534/2494, 0 new divergences.
Revise the epic's direction from a minimal Janet shim to ripping Janet
out entirely — Chez becomes the sole substrate. The missing spine: the
compiler pipeline itself only runs on Janet today (the analyzer executes
on the Janet host; the IR->Scheme emitter is host/chez/emit.janet). Phase
3 is re-scoped to self-host the compiler on Chez (emitter -> Clojure
jolt.backend-scheme, reader -> jolt-core, compile-from-source bootstrap
fixpoint). Phase 5 becomes a hard delete of both src/jolt/*.janet and
host/chez/*.janet. Sequencing: core parity first, then self-host, then
delete.
Janet's %j renders a non-ASCII char as raw UTF-8 bytes (\xC3\xA9) and a
control char / DEL as \xHH with no terminating semicolon — both forms
Chez's reader rejects (invalid character \ in string hex escape). Replace
the %j string encoder with chez-str-lit: UTF-8-decode each char and emit a
Chez codepoint escape \x<cp>;, keeping \n/\t/\r/\"/\\ and staying
byte-identical to %j for printable ASCII. Applied to every string-content
site (string/keyword/symbol/var-name/regex-source).
Unblocks the p{L} utf-8 corpus case; prelude parity floor 1532 -> 1534.
The transient assoc! accepted an odd key/val count and silently assigned
nil to the dangling key — non-Clojure, and inconsistent with the seed's
own plain assoc (which throws) and Clojure's assoc!. Make it throw.
Updates the 4 'assoc! odd args' transient spec rows to 3 :throws + 1
even-args positive, and regenerates corpus.edn. The Chez host already
threw on these, so this only realigns the corpus contract.
host/chez/dynamic-vars.ss binds the two seed-native dynamic vars that aren't
emitted into the prelude: *clojure-version* (the {:major 1 :minor 11 ...} map) and
*unchecked-math* (false). Removes their two run-corpus-prelude allowlist entries
(now 11, all passing).
*ns* is deferred to jolt-b4kl: it needs a namespace value that is not a map yet
answers (get ns :name) for the overlay ns-name, plus str/find-ns support.
Parity 1530 -> 1532/2497, 0 new divergences. emit-test 305/305.
host/chez/multimethods.ss implements the multimethod runtime: defmulti/defmethod
expand to defmulti-setup/defmethod-setup calls (+ get-method/methods/
remove-method/prefer-method/prefers). A jolt-multifn record carries its dispatch
fn and a jolt=-keyed method table; jolt-invoke dispatches it (exact match, then
isa?/hierarchy with prefer-method, then :default), reusing the overlay's
isa?/derive/make-hierarchy. The multifn's ns comes from a runtime chez-current-ns
(default user; the prelude load sets clojure.core for print-method/print-dup).
Two emit-side changes were needed:
- late-bind (:late-bind-unresolved? ctx flag, default OFF): defmulti expands to a
bare-symbol setup call, so the analyzer doesn't intern the name and a forward
reference '(area ...)' after '(defmulti area ...)' in one form was 'Unable to
resolve symbol'. The strict compiler punts these to the interpreter; the Chez
back end has none, so the flag lowers an unresolved symbol to a var-ref against
the compile ns (open-world -e semantics). Set only by the Chez make-ctx /
jolt-chez; the main compiler keeps strict resolution (host_iface late-bind?
defaults nil).
- a :var call head now routes through jolt-invoke, since a late-bound var can hold
a multifn (or keyword/coll IFn), not just a procedure. Transparent for
procedures; the hot self-recursive call is a :local known-proc, stays direct.
Class-based dispatch ((class x)/String) deferred (needs deftype/class subsystem).
Parity 1506 -> 1530/2497, 0 new divergences. emit-test 302/302. Full janet gate
green (the analyzer flag is off there; suite flakiness under parallel load only).
jolt-y1zq tail:
- 0-arg (conj) -> [] and 0-arg (conj!) -> a fresh transient vector
- nth sees through a transient (like get/count/contains?)
- irregex \p{...}/\P{...} property classes translate to the seed's ASCII char
classes (regex.ss): \p{L} -> [a-zA-Z] + non-ASCII codepoints (the seed counts
UTF-8 high bytes as letters), \p{N} -> [0-9], \p{Ps} -> [([{], etc. The
translator tracks [...] nesting so a \p{} inside a class emits its content, not
a nested class.
Two pre-existing bugs found and filed (tracked, not replicated):
- jolt-x0os: the Chez emitter mangles non-ASCII string literals into invalid Chez
hex escapes (so p{L} utf-8 crashes on the input string, not the regex).
- jolt-ea9k: the seed's transient assoc! accepts odd args and assigns nil to the
trailing key (non-Clojure; plain assoc throws). The Chez host throws
(Clojure-correct); the 4 spec rows encoding the leniency are flagged in
transients-spec.janet pending the seed fix.
Parity 1493 -> 1506/2497, 0 new divergences. emit-test 291/291.
The 1-arg map/filter/remove/take/drop/take-while/drop-while/mapcat now return a
transducer (fn [rf] rf'), and into gets a 3-arg (into to xform from). This was
the 'cdr () is not a pair' / 'incorrect number of arguments' crash bucket: the
emitter lowers (map f) and 3-arg into at an arity the native-op gate rejects, so
they fall to the value-position path and hit the bare jolt-map/jolt-into
procedure at the wrong arity. The fix is RT-side — case-lambda those procedures
plus jolt-into.
td-* factories ported from the seed (core_coll.janet); a reduced step stops the
fold via reduce-seq's existing short-circuit (inc 3n). transduce/comp/completing
are overlay and compose over these unchanged.
Parity 1467 -> 1493/2497, 0 new divergences. emit-test 278/278.
The dominant prelude-parity crash bucket was 'apply non-procedure jolt-nil':
core fns calling seed-native seq fns (core_coll.janet) that have no Chez RT
shim, so var-deref returns jolt-nil. A static scan of the assembled prelude
turned up 52 referenced-but-undefined clojure.core names.
host/chez/natives-seq.ss shims the safe seq fns over the existing seq layer:
mapcat, take-while, drop-while, partition (collection arities only — the 1-arg
transducer forms are jolt-kxsr), and sort (compare default; a comparator may
return a 3-way number or a boolean less-than). reduced/reduced? is a jolt-reduced
record in seq.ss that reduce short-circuits on and deref unwraps, so unreduced
works. identical? = jolt= (the seed's definition).
Deferred list?: a Chez lazy seq and a list are both cseq, so it can't be told
apart without a distinct list type — a real divergence risk.
Parity 1407 -> 1467/2497, 0 new divergences. emit-test 263/263.
##Inf/##-Inf/##NaN were emitted to bare inf/-inf/nan, which are unbound symbols in
Chez. emit-const now lowers them to +inf.0/-inf.0/+nan.0. The -e/element printer
renders them inf/-inf/nan (Chez's number->string gives +inf.0), and str renders the
long Clojure forms Infinity/-Infinity/NaN. assoc! is now variadic ((assoc! t k v
& kvs)) like Clojure.
Prelude parity 1382 -> 1407/2497, 0 new divergences. str of inf INSIDE a collection
still wants the long form (needs the Phase-2 recursive str renderer), so
[inf inside coll] is allowlisted. Transducer arities and the cdr-on-()/\p{} regex
gaps are split out to jolt-kxsr/jolt-y1zq.
transient/persistent!/conj!/assoc!/dissoc!/disj!/pop! as copy-on-write over the
persistent collections (host/chez/transients.ss) — each op rebuilds the persistent
coll (no in-place perf) but the semantics match, so into/frequencies/group-by work.
Adds persistent disj over pset-disj. get/count/contains? are redefined to see
through a transient (frequencies and group-by both do (get tm k) on a transient
map); vector? on a transient vector is false, which group-by relies on.
Prelude parity 1326 -> 1382/2497, 0 new divergences. emit-test exercises the direct
transient ops via run-prelude and the overlay users (frequencies/group-by/into)
end-to-end through the bin/jolt-chez -e binary.
str/subs/vec/keyword/symbol/compare/int/double/gensym — host-coupled seed natives
the overlay assumes, now def-var!'d into clojure.core via host/chez/converters.ss
(loaded last, str reuses jolt-pr-str). Semantics match the seed: str-render-one
for str (nil->"", bare chars, regex source), the 3-way core-compare port, int
truncates. The symbol no-ns sentinel is #f to match emit's quoted-symbol lowering
(jolt-symbol #f "x"), so (= 'x (symbol "x")) holds — jolt= compares ns with strict
equal?, and jolt-nil vs #f would otherwise diverge.
Prelude parity 1220 -> 1326/2497, 0 new divergences. Floor raised to 1326; three
newly-reached *ns*/var-rendering cases added to the allowlist (Phase 2). run-prelude
and the per-case program file are now PID-unique so concurrent chez runs don't read
each other's half-written files.
Emit every non-macro clojure.core form through the live analyzer -> Chez emit
pipeline as a def-var! prelude (prelude mode, tier dependency order), load it
before a user expression, and you get an -e-capable jolt-chez: analysis on Janet,
execution on Chez. driver/emit-core-prelude assembles it (each form behind a
silent load guard so the Phase-2 multimethod forms don't break the rest);
bin/jolt-chez is the -e CLI, caching the prelude on disk keyed by source hash.
run-corpus-prelude.janet is the full parity gate this opens, the prelude-backed
sibling of run-corpus-chez. First baseline: 1220/2497 evaluated cases pass, 0 new
divergences (10 allowlisted: dynamic vars, class names, eval-order — deferred
Phase 2). The rest is the punch-list: ~360 emit-fail (real host interop, out of
the analyzer subset) and ~900 runtime crashes, mostly core fns calling
host-coupled seed natives with no Chez shim yet (str/format/vec, transients).
Follow-ups jolt-t6cr/kl2l/q3w8/9ls5.
Two shims landed to get the prelude to load and run. atoms.ss: atom/deref/swap!/
reset! (+ the compare/vals kernel) — needed at load time for
global-hierarchy = (atom (make-hierarchy)). predicates.ss: the type predicates +
name/namespace/boolean the overlay assumes are seed natives, matching the seed's
strict semantics. post-prelude.ss re-asserts char?/atom? after the prelude: the
overlay implements those by reading :jolt/type, which is false for Chez-native
chars/atoms, so its def-var! would clobber the correct native versions.
Per-case Scheme files are PID-unique so a foreground -e never reads a half-written
file while the gate runs.
Closes the last clojure.core prelude emit gap (parse-uuid): the whole
non-macro core now lowers to Scheme (prelude reach 355/355).
A #"..." literal analyzes to a :regex IR node. The Chez back end emits
a jolt-regex value over irregex (Alex Shinn, BSD), vendored as the
vendor/irregex submodule -- a portable Scheme regex with PCRE/Java-style
string patterns and first-class Chez support. host/chez/regex.ss wraps
jolt's re-* surface over it: irregex-match -> re-matches (anchored),
irregex-search -> re-find, groups as Clojure [whole g1 ...] vectors,
re-seq as a jolt seq. re-pattern/re-matches/re-find/re-seq/regex? are
def-var!'d into clojure.core so prelude / -e code resolves them.
They stay OUT of the subset native-ops on purpose: irregex's
Unicode/property-class semantics differ from the seed's byte-PEG
approximation, so keeping them prelude-only avoids dragging
engine-difference divergences into the subset-parity corpus. The Janet
back end punts :regex to the interpreter (the seed compiles #"..." to a
Janet PEG), so the main language is unchanged.
Only two adaptations for Chez's top level: a cond-expand shim (Chez's is
library-only) and a normalizing error wrapper (silences irregex's 1-arg
error warnings). rt.ss load is ~0.18s.
emit-test 131/131 (regex literal + re-* parity vs the CLI oracle);
prelude reach 355/355; Chez subset 672/672, 0 divergences; full gate
green.
(.method target arg*) now analyzes to a :host-call IR node instead of
punting at analyze. The Chez back end lowers it to a jolt-host-call
dispatch for the methods the RT shims (.write -> port display,
.isDirectory -> file-directory?, .listFiles -> directory-list); any
other method stays out of subset (clean emit-time reject, so it can't
read as a compiled-but-broken corpus divergence). The Janet back end
punts ALL :host-call to the interpreter, same shape as letfn: compiles
on Chez, interprets on Janet, zero change to the main language.
Closes the io tier's print-method defmethods and file-seq: prelude emit
reach 348 -> 354/355 (50-io 20/20). The one remaining gap is the regex
literal in parse-uuid (needs a regex engine on Chez; deferred).
emit-test 122/122; Chez subset 672/672, 0 divergences; full gate green.
Closes the last two non-host-interop prelude emit gaps.
letfn now analyzes to a :let node flagged :letrec — the binding fns are bound
into the env together before any spec is analyzed, so siblings and self resolve.
The Chez back end lowers it to letrec*; the Janet back end punts it at emit
(its sequential let* can't express the mutual recursion — same interpreter
fallback as before, just decided at emit-ir instead of analyze).
(def x) with no init (declare) analyzes to a :def with :no-init instead of
punting. Chez reserves the var cell via declare-var! (which doesn't clobber an
existing root — (do (def x 7) (def x) x) => 7); the Janet back end still punts
to the interpreter, which interns a genuinely-unbound var.
fallback-zero-test now checks emit-ir too, not just analyze-form, so the real
compile-vs-interpret decision is what it asserts (letfn/def-no-init analyze but
the Janet back end punts them). letfn stays in must-punt with an updated note.
Prelude emit reach 342 -> 348/355 (40-lazy now 13/13); Chez subset 664 -> 672,
0 divergences; emit-test 110 -> 117. Full gate green.
Emit a :quote node by reconstructing the raw reader form as RT constructor
calls: symbol -> jolt-symbol, list (array) -> jolt-list, vector (tuple) ->
jolt-vector, map -> jolt-hash-map, set -> jolt-hash-set, scalars via emit-const.
The runtime value of a quote is just that literal data (the interpreter returns
the reader form verbatim).
Quote exposed a latent seq.ss bug: empty map/filter results were jolt-nil, but
Clojure's (map f []) is an empty seq, so (= () (map f [])) must be true. Return
jolt-empty-list (which seqs back to nil, so it's still a valid lazy-tail
terminator) instead — matching jolt-take/drop/rest/list.
Prelude emit reach 334 -> 342/355. Subset probe 632 -> 664/664 compiled, 0
divergences (quote + the seq fix pull 32 corpus cases into the subset). emit-test
110/110 (added 16 quote cases). corpus.edn regenerated (the 3 malformed-catch
spec rows). Full gate green.
jolt's catch is (catch class binding body*); the binding (3rd element) must be
a symbol. Neither the analyzer nor the interpreter validated it, so a non-symbol
binding crashed with an internal Janet error (expected integer key for array...)
and, in the interpreter, a malformed clause whose body never threw was silently
swallowed (returned the try value). Clojure rejects a non-class/non-symbol catch
clause; match that with an up-front error in analyze-try and eval-try.
Surfaced building the Chez try/throw emit. Regression rows in exceptions-spec
(runs x3 modes) plus a unit test asserting the clean message in interpret and
compile. jolt-kg6p.
Emit :throw as jolt-throw (Scheme raise of the raw jolt value, matching the
Janet compiled back end's (error v) — no envelope, so catch binds it directly).
Emit :try as guard (catch; the class is dropped in the IR, so it's catch-all)
plus dynamic-wind for finally. ex-info is a native-op building the tagged jolt
map {:jolt/type :jolt/ex-info :message :data :cause}, so the ex-data/ex-message/
ex-cause tier fns read it over jolt-get for free.
Prelude emit reach 303 -> 334/355 (:throw and :try gaps close). Subset probe
619 -> 632/632 compiled, 0 divergences (throw/try/ex-info pull 13 corpus cases
into the subset). emit-test 94/94 (added 11 throw/try/ex-info cases + uncaught
exits non-zero). Full gate green.
Add a prelude emit mode to host/chez/emit.janet: when emitting clojure.core
itself (not user -e), a non-native clojure.core ref lowers to a runtime
var-deref instead of being rejected as out-of-subset, so core fns chain
through each other. Default (subset) mode is unchanged — the corpus probe
still rejects unimplemented core refs for a clean signal.
core-prelude-probe.janet walks the tiers through the live analyzer->emit
pipeline and catalogs reach + gaps (macros skipped; analyze-time only).
Baseline: 303/355 non-macro core forms emit. Remaining gaps are a tight
punch-list for the next increments: :throw (29), :quote (8), :try (2), Java
host interop (6), letfn (4), declare (2). Probe has a regression floor.
emit-test 83/83 (added prelude-mode lowering assertions); subset probe
619/619 unchanged; full gate green.
emit-fn lowered multi-arity fns to a Scheme case-lambda and variadic fns to
a rest-arg lambda; the Scheme rest list is coerced to a jolt seq (nil when
empty, via list->cseq), and the named-let wrapper runs that coercion only on
first entry so recur carries the seq directly. Single fixed arity keeps the
plain-lambda fast path (fib untouched).
Also fixes a latent leak in the module-global known-procs: a throw mid-emit
(uncompilable body) left the fn's name registered, so a later corpus case
binding the same name to a keyword emitted a direct call to a non-procedure.
The cleanup now runs on the error path too. Only surfaced once the new arity
support let +24 cases compile further before hitting an uncompilable fn.
Gate: emit-test 81/81, subset probe 619/619 compiled (was 595), 0 divergences,
2036/2655 out of subset; full run-tests green (125 files).
Brings up the seq layer on the Chez runtime. host/chez/seq.ss adds one
lazy-capable node (cseq) that models Clojure's list, cons, and lazy seq -
all print as (...), all sequential-= to each other and to vectors. seq
coerces any seqable (vector/map/set/string/list/seq/nil) to a cseq or nil;
the empty seq is a distinct value printing () (rest of a 1-elem coll is ()
not nil, seq of empty is nil). Leaf ops: first/rest/next/seq/cons/list,
reverse/last, map/filter/remove/reduce/into, range/take/drop/concat/apply,
keys/vals, plus nth/peek/pop extended over seqs. map/filter/reduce apply
their fn arg through jolt-invoke, so a procedure, keyword, or collection all
work as the fn.
Dynamic IFn dispatch: a keyword/vector/coll held in a local (let binding or
fn param) and called as a fn now routes through the jolt-invoke fallback
(procedure? -> apply; keyword/coll -> lookup). The emitter only routes a
:local callee that isn't a known procedure - a named fn's self-recursion
name stays a direct call, so the fib hot path is untouched. Closes the 3
ex-known IFn divergences.
emit.janet: seq/pred ops added to native-ops with arity gates; value-position
clojure.core refs resolve to the RT procedure (native-ops names one for each),
with +/-/*// routed to flonum-coercing wrappers so higher-order arithmetic
((reduce + [])) keeps the all-double model. values.ss: cross-type sequential
=/hash so a vector and a list of the same elements are jolt= and hash alike.
rt.ss: printer learns seqs; top-level nil prints as the empty string (jolt -e
str-style). Fixed latent bug: (conj nil ...) now builds a list, not a vector.
Gates: emit-test 69/69 (fib/mandelbrot/collections/seq/IFn parity vs the jolt
oracle, fib(30) ~24ms unchanged). Subset probe 433/436 -> 595/595 compiled,
0 divergences (was 3 known), 2060/2655 out of subset. Full run-tests green
(125 files, conformance + suites included).
Broaden the Scheme back end past the numeric/functional subset to vectors,
maps, and sets. host/chez/collections.ss adds a copy-on-write persistent
vector and a bitmap HAMT (the structure 0c measured self-hostable) backing
both maps and sets, keyed by jolt-hash and compared by jolt=. emit.janet
emits :vector/:map/:set literals to the rt constructors and lowers the leaf
ops (conj/get/nth/count/assoc/dissoc/contains?/empty?/peek/pop) via the
native-ops path, with a per-op arity gate.
Also: keyword/map literals in fn position lower to jolt-get ((:k m), ({:k v} k));
arity-1 comparisons emit the vacuous jolt truth (Scheme < rejects a non-number
even at arity 1); count returns a flonum and vector indices coerce from flonum,
both consequences of the all-double number model; values.ss = / hash and the
rt printer learn collections (maps/sets render in HAMT order, so the probe
compares unordered values via =, not printed form).
Subset parity 182 -> 433/436 compiled cases (2219/2655 out of subset), 0 new
divergences. The 3 known divergences are dynamic IFn dispatch (a keyword/vector
held in a local, called as a fn) — deferred to the IFn/protocol increment and
allowlisted in the probe. emit-test 31/31, full run-tests green (125 files).
Wire the real pipeline end to end: host/chez/driver.janet boots a compile-mode
jolt ctx, runs the EXISTING Janet-hosted analyzer on actual Clojure source to
real IR, feeds it to the Scheme emitter, and runs the result on Chez. Analysis
stays on Janet (the analyzer ports to Chez in Phase 2); execution is on Chez.
emit.janet now consumes live IR (pv/phm-normalized like the Janet backend) and
covers what the analyzer actually emits, not the hand-built inc-1 shapes:
- core ops arrive as :var clojure.core/+ etc., not :rt — lowered to native
Scheme via a native-ops table (mirrors backend.janet's), `=` to jolt=.
- var cells (host/chez/rt.ss): :def -> def-var!, :var -> var-deref. Late binding
so cross-var calls (run -> count-point) and the entry crossing resolve at use.
- named fns (defn / fn self-name) bind via letrec so self-recursion resolves.
- unsupported stdlib/host refs (no core on Chez yet) are rejected at EMIT time
(clean out-of-subset signal) instead of deref'ing to nil and failing at runtime.
Number model: jolt is all-doubles (no ratios; (/ 1 2) is 0.5), so literals emit
as flonums — matches the Janet host and keeps Chez out of exploding exact
rationals (mandelbrot). jolt-num->string prints integer-valued without ".0".
Two real bugs found via the corpus probe and fixed (regression rows added):
- loop bound in parallel (Scheme named-let) but Clojure loop is sequential — a
later init must see earlier bindings; wrap a let* around the loop.
- #(...) shorthand gensyms params with a trailing `#`, invalid in Scheme — munge
it to `_`.
Gate: test/chez/emit-test.janet runs the real analyzer -> Chez for (+ 1 2),
fib(30)=832040, mandelbrot run(40), and the two regressions, parity-checked
against the Janet oracle (6/6). First parity number via the new subset probe
(test/chez/run-corpus-chez.janet, JOLT_CHEZ_CORPUS=1): 182/182 compiled corpus
cases pass, 0 divergences; 2473/2655 out of subset pending core on Chez. Full
jpm/run-tests gate green (125 files). Chez tests skip cleanly without `chez`.
Perf note (unchanged plan): emitted fib(30) ~23ms vs hand-Scheme ~5ms — the
jolt-truthy? wrapper (~3x) plus flonum (not fixnum) arithmetic, both Phase-4
type-specialization levers.
New back end half: host/chez/emit.janet consumes the host-neutral jolt IR
(ir.clj shapes) and emits Scheme, reusing the existing front-end (Option-2
backend swap). Covers the pure-functional subset: const/local/var/rt/if/do/let/
fn/invoke/def/loop/recur. Tested by hand-built IR run on Chez: (+ 1 2)=3,
fib(30)=832040, loop/recur sum=15 (4/4).
Finding: correct emit wraps every if-test in jolt-truthy?, costing ~3x on fib
(15.8ms vs hand-Scheme 5ms). Eliding the wrapper for known-boolean tests
recovers the ceiling (Phase-4 type-driven opt).
Remaining Phase 1: wire the live analyzer, var-cell late binding, RT module,
broader op coverage for mandelbrot.
0c: persistent HAMT on Chez is ~41x faster than Janet's HAMT on the collections
map-churn (258.6 -> 6.3 ms), ~15x off mutable-native (inherent persistence cost).
Decision: self-host the persistent collections in Clojure; substrate is not the
bottleneck. See docs/chez-phase0-results.md.
0a hardening: NUL-separated keyword intern key (no ns/name collision), non-finite
-safe jolt-hash. 37/37.
0a (host/chez/values.ss): Jolt value model on Chez — nil sentinel distinct from
#f/'(), interned keywords, ns+meta symbols, exactness-aware = and consistent
hash. Chez numeric tower gives ratios/bignums free. 33/33 tests.
0b (test/chez/): extract test/spec/*.janet defspec tables into corpus.edn (2655
cases, valid as both EDN and Janet data), and a runner that drives ANY jolt
binary via the CLI boundary with per-case subprocess isolation. Pluggable target
(JOLT_BIN) so the same corpus gates every host. Baseline vs Janet build/jolt:
2641/2655, 14 known CLI divergences allowlisted; gate fails only on NEW ones.
host.* replaces janet.* as the portable interop namespace (each host implements
it over its own FFI). Add a :native dep form so projects declare needed shared
libs (libcurl/openssl/zlib) — not git-fetchable, but surfaced to the user and
probed at load so a missing .so yields a precise error, not a raw dlopen fail.
Account for jolt's layered interop surface on Chez — the janet.* bridge, the
FFI-backed java.* shim libs (http-client TLS/gzip, router, db), jpm-module Janet
deps (spork/http) — with ../examples as the end-to-end acceptance gate. New
epic child jolt-cf1q.7, gated behind Phase 2.
Phased plan for re-hosting jolt's substrate on Chez Scheme, organized around two
north stars: minimal host shim (push everything possible into self-hosted
jolt-core, drop the tree-walking interpreter) and the spec/conformance corpus as
the host-neutral correctness contract. Closes obsolete Janet-backend/cgen beads
superseded by the native substrate.
Hand-translate the two compute benches into the Scheme a jolt->Chez backend
would emit, to localize the execution-substrate ceiling without porting the RT.
fib 30: 246.6 -> 5.2 ms (~47x, fixnum). mandelbrot 200: 166.3 -> 13.4 ms
(~12.4x) ONLY with flonum-specialized ops; generic float ops box every flonum
and stay ~1.7x. 13.4 ms matches jolt's JOLT_CGEN C result, so Chez's native
compiler reaches the C ceiling with no cc step, REPL intact.
Size: Chez base 2.9 MB (AOT) / 4.0 MB (dynamic) vs Janet 2.21. Memory: Chez
~32-49 MB fixed baseline vs Janet ~12 MB (the one regression). RT-bound axes
(collections/binary-trees, where Chez's generational GC should help) not yet
measured. See spike/chez/RESULTS.md.
Generic language fixes so a host-shim library (jolt-lang/http-client) can carry
clj-http-lite without baking any HTTP/native code into core:
- reader: accept #^ (deprecated metadata reader macro) as ^
- (str pattern) returns the raw regex source, not #"..." — pattern composition
via (re-pattern (str p ...)) now works
- into/conj onto a map merges map items (was (into {} [{:a 1}]) -> {nil nil})
- a Var is callable as its current value (e.g. (wrap #'f) threading a var client)
- core-class reads a :class field off a thrown table, so libraries can throw
typed exceptions whose (class e) is a JVM class name
- compiled catch unwraps the interpreter's :jolt/exception envelope (__unwrap-ex),
so a typed throw keeps its class/message across the interpret/compile boundary
- slurp accepts a byte-array; io/copy is generic over :jolt/input-stream /
:jolt/output-stream stream markers
- instance? gains a registry (__register-instance-check!) for library shim types
- new clojure.test namespace (deftest/is/are/testing/use-fixtures/run-tests with
class-aware thrown?/thrown-with-msg?)
Spec: test/spec/clojure-interop-fixes-spec.janet.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Fix four bugs surfaced by the commonmark-app example
- regex: bounded quantifiers {n,m} no longer expand exponentially. The
desugaring inlined the continuation per optional level, doubling it each
time, so {0,61} built a PEG the compiler expanded to ~2^61 nodes and hung.
Each level is now its own grammar rule referenced by name (jolt-3xur).
- strings are a seqable of chars for vec/set/into, matching seq. They went
through realize-for-iteration, which had no string case, so into/set got
raw bytes (code points) and set threw; vec used string/from-bytes (1-char
strings). realize-for-iteration now maps make-char over the bytes, and
core-vec matches (jolt-dl4s).
- clojure.string/split takes Java Pattern.split limit semantics: negative
keeps trailing empties, 0/omitted drops them (a no-match result stays
[input]), positive caps the part count with the remainder unsplit. It used
to delegate the limit to take, so a negative limit returned [] and the
2-arg form never trimmed trailing empties (jolt-ik3a).
- System/exit is registered (maps to os/exit), so (System/exit n) works
(jolt-w2wf).
* regex: single-digit backreferences \1..\9 (jolt-xtss)
\1..\9 now match the text captured by the corresponding group, so
patterns like ([-*_])\1\1 or (\w+) \1 work. The parser records which
groups are referenced; a referenced group additionally captures its text
under a tag and the backref compiles to a PEG (backmatch). Only referenced
groups change — they match possessively (the CPS-over-possessive-PEG engine
can't backtrack into a tagged capture), so backtracking back into a
backreferenced group isn't supported (rare). Unreferenced groups keep full
backtracking and position-based result capture unchanged.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
canon-key canonicalizes a collection key by re-keying a native Janet
table by the canonical form of each element/entry. canon-key returns nil
for nil, and a Janet struct can't hold a nil key or value, so a nil set
member / nil map key / nil map value was dropped during canonicalization
— #{nil 1} canonicalized like #{1} and collided as a map key. So
(count {#{nil 1} :a, #{1} :b}) was 1 and (get {#{nil 1} :a} #{1}) was :a.
Box a nested nil before it goes into the table. The marker has to be
value-hashable, not the identity-hashed mutable-table sentinel the
transients use: the canonical struct becomes a long-lived phm key, and
its hash must survive the marshal/snapshot/fork that init-cached relies
on — an unmarshalled table gets a fresh address, so its hash isn't
preserved and the map can't find its own key. An interned keyword hashes
by content. Collision risk is only a real value equal to that exact
keyword, the same negligible class as canon-key's existing set/map struct
aliasing.
The transient sentinel stays a mutable table (it's built and consumed
within one op, never crossing a marshal boundary, so identity hashing is
stable there).
Co-authored-by: Yogthos <yogthos@gmail.com>
Two paths dropped a nil set member. phs-seq read members via
phm-to-struct, whose Janet struct can't hold a nil key, so the nil was
lost on seq/print and on into-an-existing-set even though the backing phm
counted it (count/contains? then disagreed). Re-attach the nil from the
phm's has-nil slot, keeping struct-key order for the rest so set printing
stays stable.
The transient set keyed its native table by canon-key and stored the
member as the value; canon-key returns nil for nil and a nil value also
drops the entry, so conj!/disj!/contains?/persistent! lost a nil member
outright. Key by tbl-key (the same nil sentinel the transient map uses)
and box a nil value through tbl-nil-key, unboxing on persistent!.
A nil-containing set used as a map key still collides with the nil-free
one (canon-key drops nil during key canonicalization) — separate latent
bug, filed as jolt-zcm9.
Co-authored-by: Yogthos <yogthos@gmail.com>
The map build already used a transient map, but each bucket was rebuilt with
a persistent (conj (get ret k []) x) per element — an O(log n) trie path
rebuild + alloc each. A coarse grouping (few large buckets) was bound on that
conj, not the map build. Buckets are now native arrays (transient vectors,
O(1) push) frozen once; distinct keys are tracked in a side vector so the
buckets freeze in place with no second map rebuild. A bucket's first element
stays a cheap persistent [x] and only promotes to a transient on the second,
so an all-singletons grouping pays no transient alloc.
coarse (10/100 buckets, 50k): ~313ms -> ~125ms (~2.5x)
2 buckets (50k): ~322ms -> ~129ms (~2.5x)
all-unique (50k): ~949ms -> ~892ms (no regression)
Surfaced a latent bug: canon-key returns nil for a nil key and Janet tables
drop a nil key, so the canon-keyed transient map silently lost a nil-key
entry — group-by/frequencies/assoc!/into{} dropped the whole nil bucket
((group-by identity [nil nil 1]) gave {1 [1]}, not {nil [nil nil], 1 [1]}).
Route nil through a sentinel (tbl-key) at the transient-map keying sites;
persistent!/count/dissoc! work unchanged since the real [nil v] pair is kept
as the stored value, and phm already has its own has-nil slot. The transient
set has the analogous bug (needs phs nil support) — filed separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
into {}, frequencies, group-by, set, into #{} and persistent! all built
their result by folding an immutable assoc/conj per element — each call
rebuilt the O(log32 n) trie path and allocated a fresh wrapper. Add a
one-pass bottom-up HAMT builder (phm-from-pairs) and route the builders
through it, the map/set analog of the pvec bulk build in #153.
phm-from-pairs partitions entries by hash and constructs the bin/array/
collision nodes directly, with the same bin<=16 / array-node>=17 promotion
the incremental path uses — so the trie is byte-identical to one built by
phm-assoc (validated across the size and branching boundaries, including
hash collisions, duplicate keys and the nil key). persistent! map/set and
the set constructor bulk-build; into {} keeps the small-scalar-map-stays-a
-struct rule via bulk-map-from-pairs; frequencies/group-by switch to the
canonical transient form and ride the fast persistent!.
50k A/B: into {} 704->270ms, frequencies 582->160, set 615->241,
into #{} 702->240, group-by 1358->919 (bound on persistent vector conj).
Gate: conformance x3, full suite (4718 >= baseline), new maps/sets bulk
boundary specs.
Co-authored-by: Yogthos <yogthos@gmail.com>
pv-from-indexed (behind vec/mapv/filterv/into-a-vector) built a pvec by calling
pv-conj once per element — each call allocated a fresh pvec wrapper and copied
the up-to-32 tail tuple, so building an n-vector was O(n) allocations + tail
copies. Replace it with a single bottom-up trie construction: chunk the elements
into 32-wide value leaves, group nodes 32-wide up to the root, split off the
tail. The structure is identical to the incremental one — tail-offset(n) =
((n-1)>>5)<<5 is exactly the trie/tail boundary, so nth/conj/assoc/seq read it
unchanged (validated against the old builder across the size boundaries).
into-a-vector likewise stops doing a persistent pv-conj per element: it
accumulates into a native array and bulk-builds once (the transient-style path).
Measured (50k): vec 211 -> 6 ms (~36x), into [] 197 -> 15 ms (~13x). mapv is
unchanged here — it's bottlenecked on lazy map realization, not the build.
The map/set builders (into {}, frequencies, group-by, set — all HAMT-backed)
need the same bulk treatment and are a separate follow-up. Gate: conformance x3,
full suite, new bulk-boundary rows in vectors-spec.
Co-authored-by: Yogthos <yogthos@gmail.com>
Add a PNG writer so the demos produce actual images. Two pieces:
- src/jolt/png.janet — the encoder (8-bit RGB, filter None, stored/uncompressed
DEFLATE so no compressor is needed; correct CRC32 + Adler32). It lives in Janet
because per-byte work in the overlay is far too slow (a byte-array aset loop is
~30s for 360k bytes, and CRC32 over even a tiny image would be worse). Janet's
bit ops are 32-bit signed, so the 32-bit-unsigned arithmetic is done with plain
number ops (doubles hold 2^32) plus byte-level bxor. Exposed to the overlay as
janet.png/* by importing it into eval_base's module-load-env.
- src/jolt/jolt/png.clj — the jolt.png overlay wrapper: image / put! / write. The
overlay only produces pixels; the host encodes them in one pass.
mandelbrot gets a `render` subcommand (jolt -m mandelbrot render out.png [size])
that colours count-point's escape counts; the numeric-arg bench path is untouched.
Verified end to end: macOS `sips` accepts the output (so CRC/zlib are valid).
png-test covers the encoder structure (signature/IHDR/IEND) and the overlay
round-trip.
Co-authored-by: Yogthos <yogthos@gmail.com>
emit-loop compiled every loop/recur to a self-recursive local closure called once
per iteration — relying on Janet TCO for stack safety but paying a fn frame + arg
bind each iteration. The jolt-5vsp spike localized the whole ~1.43x
jolt-over-hand-Janet gap on compute loops to exactly this.
Lower instead to a Janet `while` + state vars: the loop bindings become vars
carried across iterations, a recur writes them and raises a continue flag, and a
non-recur tail value falls out through a result var. recur-name routing in
emit-recur picks the while-set lowering for loops and leaves the fn-arity self-call
path untouched.
The one subtlety is closure capture: Janet closures capture vars BY REFERENCE, so
a closure built in the body over a shared mutable loop var would see the final
value ([3 3 3]) instead of its iteration's ([0 1 2]). Each iteration rebinds the
loop names into a fresh immutable `let` before running the body, which restores
per-iteration capture. recur reads those immutable bindings and writes the state
vars, so cross-referencing args (swap, fib) need no temps.
mandelbrot 218 -> 164 ms (~11.2x JVM, from 15x). fib is unaffected — it's
fn-arity recursion, not a loop. Regression spec in control-flow-spec covers
closure capture, no-clobber recur, nested loops, sequential init, recur through
let, and that fn-arity recur still works. Gate green (conformance x3, full suite).
Note: validating this after a rebuild needs JOLT_NO_DEPS_CACHE=1 — the deps-image
cache keys on the version string, not build identity, so it served stale codegen
(filed separately).
Co-authored-by: Yogthos <yogthos@gmail.com>
Fuse an app's native-compiled numeric-leaf fns plus its source into one
static executable: no sidecar .so, no toolchain on the target. The AOT path
(#148) already produced a prebuilt module + manifest; this links them into
the jpm-built exe so the app ships as a single file.
`jolt cgen-build -m NS -o OUT` stages a build dir (src/jolt-core symlinks
into the jolt tree, a generated cg.c of the hot fns, an uberscript bundle of
the app, and an entry that bakes the runtime, installs the native fns as var
roots, and runs -main), then runs `jpm build` there — declare-native builds
cg.a and declare-executable static-links it (jpm's create-executable marshals
the module cfns and calls its static entry at startup).
Build needs cc + jpm; the result needs neither. Mechanics that bit, codified
in cgen_build.janet: stdlib_embed slurps .clj cwd-relative so the build runs
in a repo-mirroring dir; jpm hardcodes ./project.janet and sets syspath=modpath;
the executable's dofile imports cg and static-links cg.a, neither ordered nor
release-built by default, so deps are wired explicitly; cleanup must lstat (the
tree symlinks must not be followed); the inner build runs --workers=1 so it
doesn't starve siblings in the parallel gate.
test/integration/cgen-build-test.janet builds the mandelbrot fixture, runs it
from a clean dir with no src/ and no cg.so, and checks the total at native
speed. Closes jolt-a7ds.
Co-authored-by: Yogthos <yogthos@gmail.com>
Lever 1 (native codegen) is built and merged (PRs #143-148): the floor is
localized, cgen translates numeric-leaf fns to C (JOLT_CGEN, 18x on mandelbrot,
cached), and the build-time AOT path deploys native code with no cc. Replaces the
stale START HERE (which still pointed at the now-done spike) with current status
and the open work: jolt-a7ds binary fusion, jolt-v28u while-lowering, jolt-l1l4
grammar widening, jolt-qx70 hot-fn detection.
Co-authored-by: Yogthos <yogthos@gmail.com>
Splits native codegen into a build phase (needs cc) and a deploy phase (none):
- gen-c-module/compile-module compile MANY numeric-leaf fns into ONE native
module (the AOT shape), generalizing the one-fn-per-.so JIT path.
- Backend :cgen-collect? records each numeric-leaf defn's IR while the app loads
as bytecode; cgen/aot-build compiles them into one module and write-manifest
persists {sopath, [{ns name sym}]}.
- Backend :cgen-prebuilt + cgen/load-aot: the deploy run loads the prebuilt .so
(via the native builtin, no cc) and installs each cfunction as the var root
with the same timing as the JIT path, so callers direct-link to native code.
- toolchain-available? no longer crashes when cc is off PATH (os/execute raises
on a missing exe) — a toolchain-less target now gets false.
Proven end-to-end in two processes (spike/native/aot-demo.janet): build with cc,
then deploy with cc removed from PATH -> count-point still native, mandelbrot
3288753 at 12.4ms (full 18x). Test: test/integration/cgen-aot-test.janet. Default
path unchanged; the modes are opt-in. Gate green (118 files).
Remaining for a literal single binary: fuse the .so + manifest into the jpm exe.
Co-authored-by: Yogthos <yogthos@gmail.com>
compile-fn now keys the .so on a hash of the generated C + the Janet ABI + the
platform, in a persistent cache dir (default jolt-cgen under TMPDIR, override
with JOLT_CGEN_CACHE_DIR; JOLT_CGEN_NO_CACHE=1 forces a rebuild). cc runs only on
the first build of a given fn; later runs with the same source reuse the cached
.so, so the per-startup compile cost is paid once.
mandelbrot 100 whole-process wall: cold ~0.71s -> warm ~0.21s (the ~0.5s cc
cost). These cache knobs don't shape output, so they stay out of
ctx-shaping-env-vars (same as the image-cache knobs). Test asserts the .so is
content-addressed and a second compile hits the cache without the source .c.
Co-authored-by: Yogthos <yogthos@gmail.com>
Wires src/jolt/cgen.janet into the backend's :def emit. With JOLT_CGEN=1 (off by
default, needs direct-linking), a plain defn of a numeric-leaf fn is compiled to
C at def time and the cfunction installed as the var root, so direct-linked
callers embed native code. The fn is not inline-stashed when cgen fires —
callers must call the C fn, not inline the bytecode body. ^:redef/^:dynamic stay
bytecode.
The leaf-first rule falls out: run calls count-point (a user var), so run isn't a
numeric leaf and stays bytecode, calling the native count-point over the cheap
forward crossing. mandelbrot 200: 224ms -> 12.4ms (~18x), result unchanged.
Adds JOLT_CGEN to ctx-shaping-env-vars (rides the disk-cache key) and :cgen? to
resolve-run-mode. Default path (cgen off) is a no-op: cgen-root returns nil and
the normal bytecode emit runs. Gate green (117 files). Test:
test/integration/cgen-pipeline-test.janet.
Co-authored-by: Yogthos <yogthos@gmail.com>
First slice of the native-codegen tier. A new standalone module, src/jolt/
cgen.janet, that translates a numeric-leaf fn (numeric in/out, body uses only
native-op arithmetic + loop/recur/if/let/do) to a Janet native C module: params
unboxed to C doubles at entry, loop/recur lowered to a while loop, reboxed at
return. compile-fn runs cc and loads the .so via the native builtin, returning a
cfunction; it returns nil for non-candidates or when the toolchain is absent.
count-point compiles and matches the bytecode fn across the mandelbrot grid
(test/integration/cgen-test.janet, which skips the behavioral leg where cc/janet.h
are missing). Nothing wires this into the default compile path yet — detecting
hot fns and installing the C version onto the var cell is the next step.
See docs/foundational-runtime-lever1-native-codegen.md for the ceiling
(native-C ~18-22x faster than bytecode, edges out JVM) and the leaf-first rule.
Co-authored-by: Yogthos <yogthos@gmail.com>
Probes the ceiling and incremental strategy for compiling hot fns to native C,
the only lever that moves the ~10.8x Janet-VM floor the localization spike found.
Native-C mandelbrot (Janet native module) runs ~10-12ms — faster than JVM
Clojure (14.2ms) and ~18-22x faster than jolt's 219ms. The boundary cost is
asymmetric: a bytecode loop calling a C hot-fn 40k times is nearly free (~11ms),
but a C fn calling back into bytecode via janet_call costs ~3.5us/call (~152ms,
no win). So the strategy is leaf-first / whole-hot-cluster compilation, crossing
only at cold edges. A plain cc-built .so (no jpm) loads at runtime via require at
full speed, so the native tier fits jolt's dynamic compile model.
Adds the spike artifacts under spike/native/ and the writeup. Next step is
jolt-ihdp (IR->C for the numeric subset). No source changes.
Co-authored-by: Yogthos <yogthos@gmail.com>
The jolt-vs-hand-Janet-vs-JVM mandelbrot comparison splits the 15.4x floor
into two layers: a Janet-VM floor (~10.8x JVM, optimal while-loop Janet over
unboxed doubles — only native codegen moves it) plus a ~1.43x jolt loop-
lowering overhead on top. The overhead is entirely the loop/recur -> recursive-
closure-called-per-iteration lowering; hand-Janet written the same way matches
jolt, while a while+var/set version is 1.43x faster. So a cheap backend win
(jolt-v28u) sits above the structural native-codegen lever.
Adds the spike artifacts under bench/ and the results writeup; marks the spike
done in the handoff. No source changes.
Co-authored-by: Yogthos <yogthos@gmail.com>
The targeted-specialization work (jolt-ffn) concluded that the constant-factor
gap vs JVM is structural, not per-form: three targeted passes (field-read,
inline cache, ctor descriptor-bake) all came back flat. mandelbrot (pure
compute) is ~15x off JVM and that's the floor — Janet bytecode VM + mark-sweep
GC + indirect calls.
This doc hands off the successor epic (jolt-5vsp): the foundational levers
(native codegen, GC-pressure reduction, deeper devirt+inline) and, importantly,
the spike to run first — localize the 15x floor by comparing jolt-compiled vs
hand-written-Janet vs JVM mandelbrot before committing to any big lever. Also
records what not to repeat.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method reads its fields through the generic guarded keyword lookup
because the method's `this` param is untyped. defrecord now hints `this` with
the record type, the per-form inference seeds ^Record-hinted params (the
:fn branch previously typed all params :any — only the whole-program path
seeded phints), and run-passes feeds the inference the record shapes. So a
hinted param's field reads bare-index instead of going through the :jolt/type
tag guard.
This needed a with-meta fix: (with-meta sym ..) returned a proto'd table, so
symbol? was false and the macro-attached hint broke fn destructuring. Symbols
now carry metadata in-place in their struct (matching how the reader attaches
^hint), keeping symbol? true, as in Clojure.
Modest on dispatch (~3-5%): the field read is a small fraction of a dispatch;
the machinery (record-tag + protocol lookup + wrapper) dominates, which is the
inline-cache target (jolt-ez5h). But it's a correctness fix and lets any
^Record-hinted code — not just methods — drop the field-read guard per-form,
not only under whole-program.
Co-authored-by: Yogthos <yogthos@gmail.com>
The ray tracer is compute-bound and the three existing benches only cover
alloc / megamorphic-dispatch / collections. Add three axes the epic needs to
judge itself holistically:
- mono-dispatch: monomorphic protocol dispatch. Its jolt/JVM ratio (~110x) is
*worse* than megamorphic (~76x) — the JVM inline-caches a runtime-monomorphic
call site to near-free while jolt does a full registry dispatch (devirt only
fires on statically-proven receivers). Points at the call-site inline cache.
- mandelbrot: pure float compute, no alloc/dispatch. The floor at ~15x — native
arith already gets close to the JVM.
- fib: recursion, call + integer-arith overhead.
run.sh gains JVM=1, which runs each bench on JVM Clojure too and prints the
jolt/JVM ratio. collections sized up now that the map is a HAMT (jolt-684u).
README documents the axes and the current scorecard.
Co-authored-by: Yogthos <yogthos@gmail.com>
run-tests.janet runs the same file set as `jpm test` across a pool of worker
processes (one `janet FILE` each, ev-based). The full gate goes from ~790s
serial to ~98s here (8x), and more on CI where the heavy files don't thrash on
swap. CI and the docs point at it; `jpm test` still works serially.
Three things dominated the wall:
- Nine integration tests cold-built a compile ctx (~8s each); switch them to
api/init-cached so they share the prebuilt image. The cache key already
fingerprints the ctx-shaping env vars, so the direct-link ones share one DL
image and the rest share the plain one.
- core-bench's main ran on every gate (~35s of benchmark loops that assert
nothing); gate it behind JOLT_BENCH=1.
- cli-test spawned `janet src/jolt/main.janet` ~20 times at ~8s cold each
(340s under parallel load, and it was the whole wall); prefer build/jolt
(~20ms baked ctx) when present, fall back to from-source for an unbuilt tree.
type-check-test stays on cold init: a snapshot-loaded ctx loses the success
checker's op/msg detail (jolt-vley). jolt-pria tracks caching from-source
startup generally, which would let cli-test drop the build/jolt preference.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Bind *command-line-args* after the deps-image cache swap (jolt-4mui)
Under whole-program (deps-image cache active), `jolt -m NS ARG` dropped ARG:
run-main set *command-line-args* on the current ctx, but a cache HIT then
replaced ctx with the saved image (via `set ctx cached`), whose *command-line-
args* was whatever got baked when the image was saved. The stale binding won at
`(apply NS/-main *command-line-args*)`, so -main ran with the wrong (usually
default) args — silently, for any optimized -m program.
Move set-command-line-args to AFTER the cache swap so it binds on the final ctx.
Repro/regression in deps-cache-args-test.janet: first run builds the image
(arg "first"), second run (cache hit) must echo "second", not the baked "first".
* docs: RFC 0003 — phm is a HAMT, sorted colls a red-black tree
The transients RFC described phm as "bucket-based copy-on-write" and mused about
"if it ever becomes a HAMT" — it is one now (jolt-684u), and sorted maps/sets are
a red-black tree (jolt-0hbr). Update the deviation/future-work notes accordingly.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Sorted collections were a sorted VECTOR — insert-at = (into (conj (subvec es
0 i) x) (subvec es i)) is O(n) per assoc with a large constant, so building was
O(n^2): 2000 entries took 55.6s.
Replace the rep with a red-black tree (assoc/dissoc/get/contains O(log n)),
ported from the ClojureScript PersistentTreeMap (cljs.core: tree-map-add /
balance-left / balance-right / tree-map-append / balance-*-del). This tier (25)
loads before 30-macros so deftype isn't available; a node is a plain vector
[color k v left right] and cljs's BlackNode/RedNode methods become functions —
the algorithm is unchanged. A sorted-set stores elements as keys with a nil
value; its ops project the key.
The seed read the old :entries vector directly for equality/printing; route
those through a new :entries op that materializes ascending from the tree
(core_types/sorted-entries-arr + main.janet's printer).
2000 sorted-map assocs: 55.6s -> 0.98s (57x); now O(log n) (per-op cost flat
from n=2000 to 10000). Correctness in test/integration/sorted-rbtree-test.janet
(shuffled insert ordering, delete rebalancing, custom comparator, comparator
lookup, subseq, count); sorted specs + full gate green. (key/val on sorted
entries stays a pre-existing gap — entries are pvecs not host tuples; jolt-jk23.)
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add benchmark suite for alloc/dispatch/collection workloads (jolt-1r86)
The ray tracer is float-compute-bound (devirt, alloc removal, type-proving all
measured flat on it), so it can't validate the optimization passes. Add a small
cross-language suite (AWFY + CLBG style, portable Clojure) isolating the axes it
misses:
binary-trees allocation / GC pressure (escaping short-lived records)
dispatch megamorphic protocol dispatch (~1M dispatches/s; WP can't devirt)
collections persistent map/vector churn
bench/run.sh runs them; bench/README.md maps each to the pass it exercises.
collections immediately surfaced jolt-684u: the persistent hash map is O(n) per
assoc (flat copy-on-write bucket array, not a HAMT) — n=4000 assocs take 50s.
Invisible to the ray tracer (no maps).
* Persistent hash map: HAMT instead of O(n) copy-on-write (jolt-684u)
The map was a flat bucket array whose assoc copied the whole array every insert
(O(n)/assoc, O(n^2) to build). Compounding it, small maps are Janet structs that
only promoted to phm for collection keys — never for size — so a scalar-key map
stayed an O(n)-copy struct forever. Building a 4000-entry map took ~50s.
Two fixes, following ClojureScript's design:
- phm.janet is now a HAMT (hash array mapped trie): BitmapIndexedNode /
ArrayNode / HashCollisionNode, 32-way, 5 hash bits per level, structural
sharing — assoc/dissoc/get are O(log32 n). Translated from cljs.core, adapted
to Janet's 32-bit bit-ops (the hash is carried unsigned, the level index is
extracted with arithmetic, and bits are tested with band against 1<<i since
brushift rejects negative bitmaps). The public phm-* API and the value shape
(:jolt/type :jolt/phm, :cnt) are unchanged; transients are a separate rep and
untouched.
- core_coll promotes a struct map to a phm past 8 entries (not only for
collection keys), mirroring cljs PersistentArrayMap -> PersistentHashMap, so
incremental building isn't O(n^2).
20000 raw assocs: 7.1s -> 0.105s. The collections benchmark: 16.7s -> 0.2s.
Correctness covered by test/unit/phm-hamt-test.janet (oracle vs a Janet table,
nil keys, dissoc, a real hash-collision pair, and a sub-linear-assoc guard);
full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
The ray tracer is float-compute-bound (devirt, alloc removal, type-proving all
measured flat on it), so it can't validate the optimization passes. Add a small
cross-language suite (AWFY + CLBG style, portable Clojure) isolating the axes it
misses:
binary-trees allocation / GC pressure (escaping short-lived records)
dispatch megamorphic protocol dispatch (~1M dispatches/s; WP can't devirt)
collections persistent map/vector churn
bench/run.sh runs them; bench/README.md maps each to the pass it exercises.
collections immediately surfaced jolt-684u: the persistent hash map is O(n) per
assoc (flat copy-on-write bucket array, not a HAMT) — n=4000 assocs take 50s.
Invisible to the ray tracer (no maps).
Co-authored-by: Yogthos <yogthos@gmail.com>
A ^Record param hint was applied only at the final re-emit (reinfer-def), not
during the inter-procedural fixpoint. So a hinted param with no callers stayed
:any while inference ran, and a field read off it (e.g. (:origin ^Ray r)) never
told a non-inlined callee that its arg is a Vec3 — the callee's params stayed
unproven and its field reads kept the dynamic guard.
Seed declared hints as a param-type floor in the fixpoint: phint-seed (passes/
types) resolves an arity's :phints to positional record types via the
record-shapes registry, and infer-unit! initializes each fn's fresh param slots
from them instead of nil. A fixed declared type can't poison the least-fixpoint
the way an early-iteration :any would, and a hinted param now propagates its
(and its field reads') types to its callees during inference.
Scope: this closes the hinted-propagation gap. It does NOT help the ray tracer,
which uses zero ^-hinted params (only hinted fields) — its remaining type gap is
unhinted record-param inference on recursive/non-inlined hot fns, and per the
jolt-15jq A/B it's allocation-bound regardless (jolt-8flj). Tracked on the bead.
Co-authored-by: Yogthos <yogthos@gmail.com>
Dependency resolution now lives in the `jolt` CLI itself instead of a separate
jolt-deps executable. `jolt` resolves a deps.edn into JOLT_PATH/JOLT_APP_PATHS
in-process and dispatches the deps subcommands:
jolt -M:alias [args] run the alias :main-opts
jolt -A:alias CMD run CMD with the alias paths
jolt run FILE resolve, then run FILE
jolt path | tasks | task NAME
A deps.edn in the working dir is auto-resolved for the runnable commands
(repl/-m/-e/nrepl-server/FILE), so e.g. `jolt -M:nrepl` (or plain
`jolt nrepl-server`) starts an nREPL with the project and its deps loaded.
The runtime core stays deps-agnostic — it only reads JOLT_PATH. The resolver
(deps.janet) is reached only from the CLI entry and loads jpm lazily, so a run
with no deps.edn never touches it and an app baked from its own jolt/api entry
never links it. resolve-deps-argv only resolves on an explicit deps command or
when a deps.edn is present; help/version never do.
jolt-deps stays as a thin deprecation shim that forwards to `jolt`, so existing
scripts keep working. Docs (README, CLAUDE.md, building-and-deps, tools-deps)
and the help text updated.
Co-authored-by: Yogthos <yogthos@gmail.com>
scalar-replace already folds non-escaping const-key map literals
((:k {:k a ..}) -> a, and drops a let-bound map that doesn't escape).
Extend the same fold to record constructors: a (->Rec a b c) is a
positional struct whose declared field order lives in the record-shapes
registry, so a field read on a non-escaping ctor folds to the matching
positional arg and the allocation disappears.
Direct form (:field (->Rec ..)) and the let-bound form both handled,
threaded through run-passes via a per-unit shape registry (new
jolt.host/record-shapes accessor). Soundness: ctor args must be pure
(duplicated/discarded like map vals), arg count must equal the field
count, and only declared-field reads fold — a record answers the virtual
:jolt/deftype key with its type tag and any other key with nil, neither
of which is a positional arg, so those keep the allocation. pure? now
treats a record ctor of pure args as pure, so nested records (a Ray
holding a Vec3) fold bottom-up.
Allocation-bound microbench (non-escaping record built + field-read in a
hot loop): 69.6s -> 2.4s, landing on the no-record arithmetic baseline.
The ray tracer is unchanged — its vec3 results escape (returned/stored
each op), so they genuinely allocate; that's a separate problem.
Co-authored-by: Yogthos <yogthos@gmail.com>
cap truncates a deep type's field VALUES to :any so the inter-procedural
fixpoint stays finite, but it rebuilt the struct via mk-struct and dropped the
record :type tag along the way. The tag is identity — independent of field
depth — so a record stored in a deep container (a Sphere in a world vector, a
material on a hit) degraded to a plain struct, and devirtualization (jolt-41m)
and record? folding silently stopped firing on it.
Preserve :type alongside :shape when capping. Verified: a protocol call on a
record read out of a vector now devirtualizes (the call node gets :devirt-type,
which needs the receiver's record type). Sound — the tag stays accurate; only
field values below the depth cap are truncated.
No measurable wall-clock change on its own (jolt's protocol dispatch is already
cheap), but it restores the record fast path / devirt / record?-folding on
records-in-containers, and unblocks downstream work that keys off record types.
Co-authored-by: Yogthos <yogthos@gmail.com>
A bundle is closed-world — everything it needs is inlined and nothing is
required afterward — so a user defn unreachable from the entry's reference
graph can be dropped. The bundler now computes reachability from main-ns/-main
plus every non-prunable form and drops dead defn/defn- by exact source span
(formatting and reader macros in the surviving code are untouched).
Conservative and sound: only plain defn/defn- are prunable; a defn is kept if
its bare or ns-qualified name appears in any kept form, the closure runs to a
fixpoint, and any use of resolve/ns-resolve/requiring-resolve/find-var/intern/
eval/load-string disables pruning entirely. A parse failure on any file also
falls back to verbatim bundling, so the command stays as robust as a plain
concatenation. defmethod/defrecord/extend bodies are non-prunable and scanned,
so a fn reached only via dynamic dispatch stays live.
New reader/parse-all-spans returns [form start end] byte offsets so the drop
is a verbatim slice, not a re-print.
30-fn library used by a 3-fn entry: bundle 1114 -> 437 bytes (61% smaller, 27
dead fns dropped), output byte-identical.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the collection-type inference proves an argument's type, number?/
string?/keyword?/record?/nil?/some? fold to a compile-time boolean. A
const-fold now runs after inference so a folded predicate propagates and
collapses any if it gates to the taken branch.
Sound by construction: only a provable answer folds, and only when the
argument is side-effect-free (a const or local) so dropping its evaluation
is a no-op. Unknown types (:any/:truthy) and impure args keep the call.
vector?/set?/map? are left out — the :vec tag conflates a real vector with
a range/seq, so vector? could be wrong.
50M-iter loop, same shape isolated with a carry-only control: number? call+
branch 5080ms, predicate folded 1365ms — matching the 1417ms control floor,
so the 3.7x is entirely the eliminated call+branch.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Reader: #() params survive syntax-quote (auto-gensym names)
#(...) named its synthesized params with bare gensyms, so a #() written inside a
syntax-quote had its params qualified to the current ns by sq-symbol — and a
qualified symbol isn't a valid fn param. hiccup's compiler emits
`(let [sb# ..] (iterate! #(.append sb# %) ..)), which broke with "Unable to
resolve symbol: ns/_NNNN".
Name the params with a trailing # (auto-gensym suffix, like Clojure's p1__N#) so
syntax-quote maps them consistently and leaves them unqualified. Harmless outside
a backtick (just a regular symbol name).
* interop: String/valueOf static + String is a CharSequence
Two interop gaps surfaced bringing up hiccup and malli:
- String/valueOf(Object): hiccup's compiler stringifies attribute values with
(String/valueOf (or arg "")). Added the static — "null" for nil, else core-str.
- (instance? CharSequence s) returned false for a string; String implements
CharSequence, and malli's :re validator gates on it before matching, so :re
schemas always failed. instance-check now answers true for strings.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Under JOLT_OPTIMIZE a -m program run inferred + specialized EVERY loaded
namespace, including every transitive dependency. On a dep-heavy app that's
prohibitive: malli-app cold-started in ~2m10s (hundreds of dep namespaces, each
run through the per-form inline + inference passes).
The closed world a whole-program pass reasons over is the APP, not its
libraries. jolt-deps now passes the project's own source roots (its deps.edn
:paths) to the runtime as JOLT_APP_PATHS. A namespace loaded from an app root
gets full optimization (and joins the one whole-program fixpoint); a dependency
namespace compiles at default cost — :inline? off for its load, so the per-form
optimize passes don't run over library code — staying direct-linked but
generically typed (the open-world default). With no app roots declared (a bare
program run, or jolt without jolt-deps) everything counts as app, so behavior is
unchanged.
malli-app JOLT_OPTIMIZE cold start: 2m10s -> 4.5s. Compute-heavy programs whose
hot code is their own namespaces (the typed ray tracer) are unaffected — their
code is app code and still fully optimized (9s/frame render). Applied at runtime
in main for the same baked-at-build-time reason as JOLT_PATH; added to the
ctx-image cache key. Help text corrected: optimization is opt-in, not default.
Co-authored-by: Yogthos <yogthos@gmail.com>
A deftype field tagged ^:unsynchronized-mutable / ^:volatile-mutable is set!-able,
but under direct-link immutable records are shape-rec tuples, so set! errored
("Can't set! field on non-deftype: tuple").
A deftype with any mutable field now opts out of the shape-rec layout and uses
the existing :jolt/deftype table form regardless of :shapes? — set! already
mutates that form and field reads route through the tagged-table path. Such a
type is also not registered as a shape, so the inference never emits a bare-index
read against the table. Immutable deftypes/records keep the fast shape-rec.
deftype extracts per-field mutability from the field metadata and passes it to
make-deftype-ctor, which picks the representation at ctor-build time.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Don't direct-link a var redefined earlier in the same unit (jolt-wf4)
defrecord/deftype expands to (do (def R (make-deftype-ctor ...)) (def ->R R) ...),
so the ->R alias references R within one compiled unit. Under direct-link a var
ref embeds the cell's root as a compile-time constant, but on a redefine R's old
root is still in place when that unit compiles — the (def R new) sibling hasn't
run yet — so ->R sealed to the stale pre-redef ctor. (defrecord R [x a])
(defrecord R [a x]) (:a (->R 10 20)) read the old [x a] layout and returned 20.
Track the vars a unit (re)defines and force their later in-unit references to the
live indirect deref. The cell is registered only after its own init is emitted,
so a recursive self-reference inside the init still direct-links (it runs after
the def completes); only sibling references after the def go indirect.
* Emit Janet's `in` as a value so a user local can't shadow it (jolt-fjb1)
The back end emits `in` to deref var cells ((in cell :root)) and index
shape-recs. It emitted the bare symbol, so a user local named `in` shadowed
Janet's builtin in the surrounding scope and the generated cell-deref called the
user's value as a function — "<table> called with 2 arguments, possibly expected
1". malli's explainer binds [value in acc], so m/explain hit this on every
schema (m/validate was unaffected — its path doesn't bind `in`).
Embed `in`'s function value at the emit sites (as jolt-call/core-get already
are); a value in head position can't be shadowed. Fixes m/explain on malli
(loaded with JOLT_FEATURES=clj so its .cljc reader-conditionals resolve).
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
eval-form treated only a reader LIST (a Janet array) as a call; a runtime-built
list — a plist or lazy-seq from cons/concat/list or ~@ (list?/seq? true but
array? false) — fell through to self-eval. So (eval (cons '+ '(1 2))) returned
the list as data instead of 3, and a macro whose output contained such a subform
left it unevaluated. Add a plist?/lazy-seq? branch that coerces to the element
array via d-realize and dispatches through eval-list; an empty list self-evals.
The analyzer already punts these forms to the interpreter (analyze's :else ->
uncompilable -> interpreter fallback), so this one interpreter branch fixes the
correctness bug across the eval and macro-expansion paths; compiling them
directly (vs punting) would be a separate perf change. Verified: conformance
355/355, syntax-quote ~@ splice, list values unchanged.
A deftype with (Object (toString [_] s)) had its toString ignored: the generic
object-methods "toString" fired in dispatch-member before the record's own
method (the record isn't a tagged shim, so that guard passed), and str rendered
the #Type{...} data repr instead of routing through toString.
- dispatch-member: a record's own method (instance/reified/protocol) now wins
over the generic object-methods table — so .toString/.equals/.hashCode on a
record use the record's definitions; plain records still reach object-methods.
- str: add a late-bound record-tostring-cb (wired per-ctx by
install-print-method-cb!, mirroring print-method-cb) that str-render-one
consults for records — a deftype with a custom toString renders via it, plain
records keep the data repr. pr-str is unchanged.
Needed by hiccup's RawString. Adds deftype-tostring-spec (.toString + str +
concatenation + a regression guard that record-less-toString keeps its repr).
walk only handled vector?/map? and fell through :else for everything else, so
postwalk over a quoted list (a plist) never touched its elements —
postwalk-replace with symbol keys silently no-op'd, which broke
clojure.template/apply-template (found during reitit work). Add list? (rebuild as
a list) and seq? (map over it) branches after the vector/map ones so concrete
collections stay authoritative. Adds walk-spec covering list/seq walking plus a
vector/keywordize regression guard and the apply-template trigger.
The 813-line host_interop.janet (java.lang statics, java.time, the 458-line
java.io/util/net/sql/text install-io!, and collection interop all in one file)
becomes a 19-line aggregator over src/jolt/interop/:
- java_base.janet — java.lang statics (Math/Thread/System/Long) + the java.time
shim + the shared chr/pad2/formatter coercion helpers
- host_io.janet — java.io/util/net/sql/text shims; imports java_base for the
shared helpers it reuses (it constructs java.time values and
formats dates)
- collections.janet— the late-bound .iterator/.nth/.count/.seq interop hooks
host_interop.janet just loads the three and runs install!/install-io!/
install-collections! in order. chr/pad2/formatter become public (they cross the
java_base->host_io boundary now). The registry machinery (class-statics/
tagged-methods/register-*!) stays in the evaluator, which loads first — moving it
out touches the hot dot-dispatch and is left as a separate step.
Pure move, no behavior change. Also fixes a cache footgun the subdir exposed:
source-fingerprint walked src/jolt/ with a non-recursive (os/dir), so an edit to
an interop/ file would not have busted the ctx image cache (stale-image bug) —
made it recurse, keyed by repo-relative path.
Full gate green: host-interop-spec (130+ rows across every JDK area), ctx-image
cold/warm, conformance x3, suite >=4695/88, fixpoint.
init-cached (core image) and the deps-image (main.janet) each hand-wrote the same
fork->slurp->validate-ctx?->rewire load and the snapshot->tmp->atomic-rename save.
Extract load-ctx-image / save-ctx-image (in api.janet, beside snapshot/fork): the
two callers now differ only in the validity predicate they pass — none for the
core image (its source fingerprint is already in the path), a deps-manifest mtime
check for the deps image. The per-process print-method-cb rewiring an image
restore must replay is threaded as a callback so the helpers don't depend on core.
Kept in api.janet rather than a new ctx_image.janet module: Janet's `use` doesn't
transitively re-export, and snapshot/fork already live in api and are consumed via
(use ./api) by main and the test harnesses — a separate module would force every
caller to import it directly. (load-image/save-image collide with Janet builtins,
hence the ctx- prefix.)
Full gate green; ctx-image-test cold/warm + deps tests pass.
core-count/core-seq/core-conj each walked a chain of (and (table? x) (= :jolt/X
(get x :jolt/type))) predicates — re-fetching the type tag per predicate and, for
conj, a 14-deep nested if. Replace with a single (case (get coll :jolt/type) ...)
per op: the type is read once and the arm calls the concrete op directly. Host
values (tuple/array/nil) and tuple-based shape-recs carry no :jolt/type and stay
in a per-op fallback cond (shape-rec? kept before tuple? — shape-recs ARE tuples).
Factor the two map-conj paths (phm vs struct) into one conj-into-map parameterized
by the assoc fn.
Behavior-preserving: the :jolt/type tags are disjoint, so the case is an exact
re-expression of the predicate chain; the fallbacks reproduce the originals
(incl. count erroring on a raw host table, which only seq ever handled).
This is the perf-neutral realization of the planned collection vtable. A shared
:jolt/type->{ops} table was tried first and REGRESSED core-bench ~2-4% (table
lookup + indirect call on the hot path); the inline case instead runs ~1.7%
FASTER than main (4864ms vs ~4950ms baseline) since one type-get + a jump beats
the sequential predicate chain. Full gate green (suite >=4695/88, fixpoint).
analyze-try assoc'd :catch-sym/:catch-body/:finally nil-when-absent, so a try
with no catch (or no finally) carried a nil-valued key — which makes the node a
phm in jolt's map representation and forces the back end to densify it
(norm-node) before reading :op. That's the map-nil-representation trap Phase 2
already cleaned up for def/fn/arity nodes. Add those keys only when the clause is
present, matching the arity :rest discipline; a try node stays a fast struct.
Behavior-invisible: emit-try reads each key with a nil-safe (node :k) and gates
on it, so an absent key and a present-nil key are indistinguishable to every
consumer. Adds ir-try-shape-test asserting the node shape across all four
try/catch/finally combinations plus end-to-end eval.
Note on scope: the plan's "delete the defensive norm-node calls" is NOT done — it
can't be. {:op :const :val nil} (e.g. (def x nil)) and nil map keys are
inherently phm, so the emit-dispatch norm-node guards a real case, not a
present-or-absent artifact. This PR removes a source of gratuitous phm nodes
rather than the densification itself. Full gate green.
Six bottom-up IR rewrites (const-fold, inline-node, subst, flatten-lets,
subst-lookup, scalar-replace) each hand-listed every op's child positions —
~250 lines of identical "recurse children, rebuild" arms that had to be kept in
sync whenever an op was added. Extract one map-ir-children into ir.clj that
knows each op's child layout; each walk keeps only its genuine specials
(const-fold's invoke/if, inline-node's invoke, subst's local/let alpha-rename,
scalar-replace's invoke/let folds) and delegates the rest.
The combinator is total over the op set, so the walks are now total too: a
couple soundly gain coverage they previously skipped (const-fold now folds
inside :try; subst-lookup now recurses :def inits, which fixes a latent dangling
ref where a dropped const-key-map binding was referenced inside a def). These
are sound — all six are result-preserving optimizations — and 3-mode conformance
+ fixpoint confirm identical program behavior.
map-ir-children is shape-preserving for :try (recurses :catch-body/:finally only
when present, never assoc's nil) so it can't turn a struct node into a phm.
Written with cond/get only, matching the passes' tier, so no new load-order dep.
Predicates (body-closed?/pure?/local-escapes?), the type-threading infer, and the
Janet backend emit stay as-is: their conservative :else defaults / [type node]
threading / host language don't fit a node-rebuilding combinator.
Adds ir-passes-test coverage for folding reaching fn/loop/try bodies. Full gate
green (conformance x3, suite >=4695/88, fixpoint stage1==2==3, inline-sra + devirt).
eval-dot copy-pasted its entire dispatch chain across the (. obj method args...)
and (. obj member) forms — string/number/object/tagged-shim lookup duplicated,
hand-synced on every interop change. Extract one dispatch-member that takes the
evaluated args plus a has-args flag. The shared head (string/number/object/
tagged) is single-sourced; the genuinely divergent tails (call form: record →
native field → coll-interop(args); bare form: zero-arg coll-interop → field /
zero-arg method) stay branched on has-args. The guards that differed between the
arms (object-methods checks table? only; tagged dispatch checks table-or-struct;
bare-form tagged dispatch requires the member present) are preserved verbatim,
keyed off has-args, so behavior is identical.
Adds a "dot dispatch arms" spec locking the divergent cases: zero-arg vs
with-arg coll-interop, record/deftype zero-arg vs with-args methods, -field
access. Full gate green.
5d: document the seed↔overlay boundary and add a drift check. core fns split
across a Janet seed (core-X, registered in core-bindings) and a Clojure overlay;
five names (char?/sorted?/sorted-map?/sorted-set?/transduce) carry a defn in
both, with the overlay copy authoritative and the seed copy internal-only. The
into-vs-transduce home asymmetry was undocumented. Adds docs/seed-overlay-registry.md,
SEED-TWIN: comments at the five seed sites, and a build-time drift check
(test/unit/seed-overlay-registry-test.janet) that recomputes the twin set from
source and fails if it diverges or a twin leaks into core-bindings.
5e: rep↔API pointer comments in pv/plist/phm/phs/lazyseq (representation lives
here; Clojure-facing ops dispatch in core_coll/core_types) and back-pointers in
core_coll. No behavior change — comments, docs, one source-analysis test.
Full gate green (suite ≥4695 pass / ≥88 clean files), drift check passes.
main.janet held ~45 lines of env-knob policy (open-mode / direct-link / optimize
/ shapes / whole-program gates) that couldn't be unit-tested without the CLI, and
two disk-image caches (api/init-cached, main/deps-image) each hand-built a
POSITIONAL "%q|%q|..." key that silently misaligned if a ctx-shaping knob was
added in only one place.
config.janet now owns:
ctx-shaping-env-vars the canonical list of env vars that shape the built ctx
ctx-cache-key a labeled key (name=value) over a prefix + every shaping
var, so adding a knob updates BOTH cache keys at once and
can't positionally alias two different builds
resolve-run-mode [open-mode? main-entry?] -> the ctx env knob map
main shrinks to: compute open-mode?/main-entry? from argv, call resolve-run-mode,
install the knobs. Both deps-image-path (main) and image-cache-path (api) build
their keys via ctx-cache-key. New test/unit/config-test.janet locks in the
run-mode cases and asserts every ctx-shaping env var participates in the key.
Scope: this is 4a + the 4b cache-key footgun fix. The optional 4b cleanup
(folding the load/save image dance + aot marshal helpers into one ctx_image
module) is left for a follow-up — it's lower value and higher blast radius.
No behavior change (cache keys now key on a superset of env vars, so at worst a
one-time cold rebuild). Gate green: conformance 355x3, clojure-test-suite 4718
pass (>= 4695 baseline), config-test, full jpm test exit 0.
Two small structural dedups from the plan's Phase 3c.
reader.janet: read-list/read-vector/read-set each had a copy of the same
read-loop (skip whitespace, stop at close char, drop #_ discards, splice #?@).
The skip/splice logic had drifted between them once. Hoisted into one
read-delimited [s pos close err] -> [items end]; the three readers now just wrap
its result. read-map keeps its own loop (its key/value pairing needs a different
value-slot scan).
phm.janet: phm-bucket-find/contains?/assoc/dissoc and phm-get each open-coded the
same stride-2 key scan. Extracted bucket-index-of [bucket k] -> index|nil; all
five now share it.
No behavior change. Gate green: conformance 355x3, clojure-test-suite 4718 pass
(>= 4695 baseline), full jpm test exit 0.
types.janet held five concerns under one generic name. Split into sibling
modules along its existing section boundaries:
types_symbols characters + symbol helpers
types_var Var
types_ns Namespace
types_ctx Context (+ inst/uuid values)
types_protocols protocol/type registry + shape-records
types.janet is now a pure aggregator: it loads the clusters in dependency order
and re-exports their defs (import :prefix "" :export true), so every consumer
keeps its single (use ./types) unchanged.
Order-preserving (statically verified: zero backward references, zero
cross-cluster private helpers — the cleanest of the seed splits). No behavior
change.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
evaluator.janet was a 2597-line file with a 680-line eval-list. Split into
cluster modules behind a re-export aggregator (same pattern as core):
eval_base forward vars, syntax-quote, ns-loading, registries, jolt-invoke
eval_resolve symbol/var resolution, params, destructuring, class lookup
eval_runtime protocols, multimethods, deftype/reify, install-stateful-fns!
eval_special the special forms (eval-list dispatch)
evaluator.janet stays the module every consumer imports: it loads the clusters
in dependency order and re-exports their defs (import :prefix "" :export true),
so the five (use ./evaluator) consumers are unchanged. It still owns the
eval-form entry that ties resolution + special forms + map/coll evaluation.
In eval_special, the giant eval-list match is exploded: each multi-line arm is
now a named (defn eval-<form> [ctx bindings form] ...) — eval-def, eval-fn*,
eval-let*, eval-loop*, eval-try, eval-set!, eval-dot, etc. — and eval-list is a
thin dispatch table over them. "where is try handled" is now `grep eval-try`.
Order-preserving (statically verified: no symbol used before its cluster loads;
zero backward refs). 27 helpers shared across clusters are now public so `use`
shares them. The two near-duplicate .method dot blocks are NOT merged here — that
is a behavior-sensitive dedup tracked separately (jolt-eos3); this PR is pure
moves + the mechanical eval-list explosion, no behavior change.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
core.janet was a 3013-line grab-bag. Split into six order-preserving cluster
modules:
core_types vector helpers, predicates, math, comparison, equality
core_coll collections, transducers, seqs, HOFs, constructors
core_print string + pr-str/str rendering
core_io I/O, files, JDBC, compare, type
core_refs arrays, bit ops, coercions, hash, atoms/refs
core_extra additional clojure.core fns, transients, hashing
core.janet stays the module everyone imports: it loads the clusters in
dependency order and re-exports each one's defs (import :prefix "" :export true),
so every consumer keeps its single (use ./core) with no change. The aggregator
also owns core-bindings and init-core!, which reference fns from every cluster.
The split preserves definition order exactly (verified: no symbol is used in a
cluster that loads before its definition), so seed load-order semantics are
unchanged. Three private helpers used across clusters (map-entries-of,
map-assoc1, str-render-one) are now public so `use` shares them; the five
forward vars (canon-key/jolt-equal?/pr-render/core-compare/print-method-cb) each
stay within their owning cluster.
No behavior change. Gate green: conformance 355x3, clojure-test-suite 4718 pass
(>= 4695 baseline), full jpm test exit 0.
passes.clj was a 1486-line grab-bag mixing three weakly-coupled concerns. Split
along the clusters the review mapped (only run-passes + the dirty flag were
shared):
jolt.passes.fold const-fold + the shared scalar-const? predicate (base)
jolt.passes.inline inline + flatten-lets + scalar-replace
jolt.passes.types collection-type inference + success checker + driver API
jolt.passes façade: run-passes + :refer re-exports of the driver fns
the back end looks up by name
scalar-const? was used by both the inline pass and the inference walk, so it
moves to fold (the base layer) and both refer it. The check-mode state stays
private to jolt.passes.types behind a new run-inference fn; run-passes calls it.
build-compiler! loads the three in dependency order before the façade, mirroring
the existing jolt.ir -> jolt.analyzer bootstrap. No behavior change. Also fixed
the stale ns docstring that listed four passes and omitted the type system.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
Completes the phm.janet decomposition (jolt-bvek): after lazyseq left, the set
follows. phm.janet is now purely the PersistentHashMap; phs.janet is the thin
set layer over it (members are keys -> true), and (use ./phm) for the builders.
Importers using set?/phs-* via (use ./phm) add (use ./phs); backend's emitted
set literal head changes phm/make-phs -> phs/make-phs. Behaviour unchanged
(sets verified interpreted + compiled; full gate green).
phm.janet held the PersistentHashMap, the PersistentHashSet, AND the LazySeq
primitives — a lazy sequence has nothing to do with hash maps; both were just
tagged tables, which is why they shared a file (jolt-bvek). An agent looking for
lazy-seq realization would never grep phm.janet.
Move the LazySeq section (lazy-seq?/make-lazy-seq/realize-ls/ls-first/ls-rest/
ls-rest-cached/ls-seq/ls-count/lazy-cons) to a new self-contained lazyseq.janet
(janet builtins only, no jolt deps). Importers that used the fns through
(use ./phm) add (use ./lazyseq); host_interop's one phm/lazy-seq? becomes
lazy-seq?. Behaviour unchanged (covered by test/unit/lazy-seq-test.janet + the
full gate). phs split is a follow-up.
Co-authored-by: Yogthos <yogthos@gmail.com>
Cheap-version consolidation of the host-interop sprawl (jolt-jx5l). The JVM
class/method shims were scattered across javatime.janet, evaluator.janet,
api.janet and core.janet; the recent hiccup/markdown/malli fixes landed ad hoc.
- Rename javatime.janet -> host_interop.janet. The name described ~20% of its
contents (java.io/util/net/sql/lang all lived there); it is now the one home
for host shims, and greppable.
- Move the four hardcoded static tables (Math/Thread/System/Long) out of the
evaluator and register them through the generic class-statics registry. The
special-case dispatch in resolve-sym is deleted — one mechanism, not two.
- Move the collection-interop wiring (set-coll-realizer!, set-coll-interop!,
malli's LazilyPersistentVector/PersistentArrayMap statics) from api.janet into
host_interop's install-collections!. api's 40 lines of wiring become a single
(import ./host_interop).
No behavior change. Full per-package src/jolt/interop/ split is the follow-up;
string/number/object method tables and core's File/JDBC ctors stay put for now
(they're coupled to the dot-dispatch / collection layer). Full gate green.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Protocol/interop fixes to run metosin/malli
Bringing up malli (schema validation) surfaced a batch of protocol and host-interop
gaps. m/validate now works across the schema vocabulary (predicates, :map incl.
nested/optional, :vector, :tuple, :enum, :maybe, :and, bounded int/string).
- extend-type and reify now accept MULTIPLE protocols in one form (each bare
symbol switches the current protocol). reify records every protocol it
implements, so instance?/satisfies? recognise all of them.
- Protocol method params support destructuring: reify/extend-type/deftype/
defrecord emit (fn ...) (which desugars patterns) instead of raw fn*.
- instance? of a PROTOCOL works like satisfies? for reify/record instances,
matching short names across qualified/bare protocol references.
- @x reads as the qualified clojure.core/deref, so it still derefs where a ns
excludes and rebinds deref (malli does). Updated reader-test + the reader
spec/grammar (S11, deref rule).
- Java collection interop on jolt collections: .nth/.count/.valAt/.get/.seq/
.containsKey route to the clojure.core equivalent (1-arg and 0-arg paths).
- java.util.HashMap capacity/load-factor constructors + .putAll.
- A class used as a value resolves to its instances' type, so Pattern -> the
regex type (malli keys class-schemas by it).
- Shims for malli's load path: LazilyPersistentVector/createOwning and
PersistentArrayMap/createWithCheck statics.
m/explain not yet working (jolt-fjb1). Full gate green.
* satisfies? recognizes reify, consistent with instance?
A reify's protocol methods are instance-local, so they aren't in the global type
registry that type-satisfies? consults — satisfies? returned false for a reify
even when it implemented the protocol. Check the protocols the reify records on
itself (the same :jolt/protocols list instance? uses), matching short names like
instance? does. Covers single- and multi-protocol reify.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add architecture refactor plan
Synthesizes a six-part architectural review into phased, gate-validated cleanup
work. Targets LLM-maintainability: one home per feature, no god-files, explicit
checked contracts, no copy-paste dispatch. No code changes yet — the plan only.
* Refactor phase 0: dead code + isolated bugs
Pure cleanup ahead of the structural phases (docs/architecture-refactor-plan.md).
No behavior change except the two bug fixes, which are covered by a regression row.
Dead code (all verified zero-reference or overridden):
- core-resolve / core-satisfies? / core-type->str seed stubs + bindings —
resolve and satisfies? are interned by install-stateful-fns! (the seed copies
were shadowed); type->str was an inert SCI stub with no callers.
- find defined twice in 20-coll.clj; the dead copy returned a plain vector
(wrong — the live def at :787 returns a real map-entry) with a comment that
contradicted it.
- mark-hint (passes.clj), phs-to-struct (phm), shape-vals / ns-imports-fn
(types) — unreferenced.
- redundant local pad2 in javatime (module-level one already in scope).
Bugs:
- File.toURL stored :url but every :jolt/url method reads :spec, so a URL from
(.toURL file) returned nil from all its methods. Now stores :spec (+ spec row).
- pl-rest had a no-op (if (plist? r) r r); collapsed to r.
- :map-shapes? was missing from the deps-image cache key — two runs differing
only in map-shapes could reuse each other's image.
Also dropped read-quote's unused pos param. Full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Bringing up yogthos/markdown-clj surfaced a batch of Clojure-conformance gaps:
- clojure.java.io/writer returned nil for a Writer/StringWriter (it only
handled paths); now passes a Writer or file handle through, like reader does.
- StringWriter had no :close field, so with-open errored closing it.
- java.io.Reader had no .readLine method (only the :read-line-fn used by
line-seq); markdown's main loop calls .readLine directly.
- Writer.write(int) wrote the int's digits instead of the char for that code.
StringBuilder.append(int) keeps Java semantics (the digits) — the two differ,
so the char-code path is local to the writer, not shared render-piece.
- drop-while over a string errored in array/slice; it now char-seqs the string
like take-while/remove already do.
- re-seq returned an empty seq instead of nil on no match, so
(if-let [m (re-seq ...)] ...) always took the truthy branch — an infinite loop
in markdown's thaw-string.
- The #() reader didn't scan % inside map {} or set #{} literals, so
#(identity {:text %}) compiled as a 0-arg fn.
re-seq-nil and the #() map/set scan are general bugs, not markdown-specific.
Co-authored-by: Yogthos <yogthos@gmail.com>
Some Clojure libraries loop with the Java Iterator protocol — e.g. hiccup's
iterate! does (let [it (.iterator coll)] (while (.hasNext it) (f (.next it)))).
jolt had no Iterator, so (.iterator coll) returned nil and the loop did nothing
(silently dropping content). Add an (.iterator coll) object-method that returns a
:jolt/iterator over any seqable, with hasNext/next; the collection is materialized
via core's realize-for-iteration (late-bound through the evaluator since core
loads after it, wired in api). Found while bringing up weavejester/hiccup.
Co-authored-by: Yogthos <yogthos@gmail.com>
Two things made a program run catastrophically slow to start (ring-app: 111s),
which is backwards — direct-linking should make startup FASTER (Clojure: smaller
classes, faster startup; Stalin: whole-program analysis is strictly ahead-of-
time, zero startup cost). We were doing the opposite: paying compile-time work at
every startup.
1. Decouple inference from direct-linking. Direct-linking (cheap compile-time
call resolution) was bundled with the whole inference/specialization pipeline
(inlining + scalar replacement + structural types + the per-ns re-emit
fixpoint) — the expensive part. Now a program run direct-links + uses records
by default (fast, faster calls) but the inference is opt-in via JOLT_OPTIMIZE
(or an explicit JOLT_DIRECT_LINK / the native build). ring-app: 111s -> 6s.
2. Compile dependencies once. The baked core loads in ~10ms, but a program still
re-compiled every dependency namespace (reitit, ring, selmer, honeysql, …)
from source on every run — the whole 6s. Snapshot the ctx after the require
chain to a per-project image (the same marshal/fork the core image uses),
keyed on version + entry + source roots + flags, with a per-file mtime
manifest so any edit invalidates it. First run compiles + caches; later runs
fork the image (~10ms) and skip compilation. ring-app: 6s -> ~1s warm,
competitive with the JVM. JOLT_NO_DEPS_CACHE disables.
Full gate green (all -m integration tests now exercise the cache); deps-image
invalidation verified (touching a source recompiles).
Under direct-linking a record is a Janet tuple (its shape-rec), and core-vector?
just delegated to jvec? which is true for any tuple — so (vector? a-record) and
(sequential? a-record) returned true. That broke map-destructuring of a record:
the destructure coerce treats a sequential source as & {:keys} kwargs and does
(apply hash-map x), so destructuring a record fed its entries to make-phm as a
flat kv-list and corrupted. Surfaced as reitit's router crashing on a wildcard
route ('expected integer key for tuple in range [0,5), got 5') whenever the new
direct-link default was on; minimal repro is (let [{:keys [a]} (->R ...)] ...).
Fix: core-vector? excludes shape-recs, matching Clojure (a record is not a
vector or sequential). jvec? is unchanged for internal representation dispatch.
Regression cases added to record-declared-shape-test.
Running a program is a closed world — every namespace is required, then it runs
to completion — so make it direct-link by default (inlining, record shapes, the
inference's specialization), and for a -m/-M entry auto-enable the whole-program
cross-namespace inference pass. A decomposed multi-namespace program was ~3.7x
slower than the same code in one namespace purely because per-namespace
inference can't see a caller in a not-yet-loaded namespace; this closes that for
the common case with no flags and no hints.
Interactive modes (repl, -e, nrepl-server) stay indirect/open — they have to let
you redefine vars, which direct-linking seals against. Opt-outs:
JOLT_NO_DIRECT_LINK forces the open path even for a program run (hot-reload,
runtime redefinition); JOLT_NO_WHOLE_PROGRAM keeps direct-linking but per-ns;
JOLT_DIRECT_LINK / JOLT_WHOLE_PROGRAM still force-on. Namespaces required inside
-main (after the batch pass) fall back to per-ns inference.
The success checker (RFC 0006) rides on the inference for free, but a casual
program run shouldn't spam type warnings just because it now direct-links, so its
default-on is suppressed when direct-linking was auto-enabled (:direct-link-auto?);
an explicit JOLT_DIRECT_LINK or JOLT_TYPE_CHECK still turns it on. whole-program-
test and devirt-test opt their per-ns baseline out of the new auto-default.
Docs: RFC 0005 gains 'Compilation modes and defaults' + 'Cross-namespace
inference'; RFC 0004 documents cross-ns/param hints; self-hosting-compiler and
--help updated. Full gate green.
A ^RecordType hint only resolved against the current namespace's ctor key, so a
hint naming a record defined in another namespace degraded to :any. That made a
decomposed multi-namespace program much slower than the monolith: per-namespace
inference can't see a record param's callers in other namespaces, and the
declared hint that could have typed it was dropped.
Resolution now works cross-namespace, for both record FIELD hints (defrecord)
and fn PARAM hints, in both spellings — ^Vec3 where the type is referred and
^v/Vec3 where the namespace is aliased:
- reader keeps a tag's namespace qualifier (^t/Ray -> "t/Ray", was "Ray").
- make-deftype-ctor-impl indexes each ctor closure by value; record-hint-ctor-key
resolves a hint name against the COMPILE ns (referred names live there; aliases
resolve through it) and maps the type var's root back to its home ctor key.
Using the ctor value, not the var's :ns, is what makes :refer work — :refer
re-interns a fresh var whose :ns is the referring ns.
- the analyzer captures record param hints as arity :phints [name ctor-key];
reinfer-def seeds those param types, so a record param is typed even with no
inferred caller — the open-world / cross-ns case.
Effect on the multi-namespace ray tracer: per-ns compile 30.4s -> 7.9s with
param hints, matching whole-program (8.1s) and the single-ns monolith (8.3s).
cross-ns-hints-test covers field + param hints, refer + as, and the reader tag.
direct-var? now treats a cfunction root the same as a function root, so a
call/ref to a native fn (clojure.math/sqrt et al.) embeds the value instead of
a per-call cell deref. This was the hot indirection in the ray tracer — sqrt
runs every bounce — and it applies in every direct-link build, not just
whole-program.
const-link? is new and whole-program-only: in a closed world every non-dynamic
var has a stable root, so embed it as a constant (quoted unless it's already
callable) rather than reading the cell each reference. Covers what direct-var?
can't — ^:redef vars (reloading is off under the flag), data defs, and record
type/ctor roots. Dynamic vars stay indirect; a nil (not-yet-defined) root stays
indirect and the whole-program re-emit picks it up once the root is in place.
Measured on the records ray tracer: hot-path indirect refs (sqrt + data vars)
gone; the only indirect refs left are cold defrecord self-references. whole-
program-test now also checks a ^:redef fn and a data def so the per-ns vs
whole-program comparison guards const-link soundness.
A record field can carry a type hint — ^Vec3 (a defined record type) or ^:num —
and the inference now resolves it so reading the field back yields that exact type
instead of :any. A Vec3 stored in a Ray field reads out as Vec3, so the vec ops on
field-read values prove their reads (bare-index). This is Stalin's per-slot type
sets, but DECLARED rather than inferred: the exact shape is known up front.
- deftype captures each field's :tag / :num metadata (was stripped) and passes it
to make-deftype-ctor; the ctor registers per-field tags, resolving a record-type
hint to its ctor-key (same-ns) so the inference can look it up directly.
- call-ret-type builds a record's struct type with field types resolved from the
hints, recursing into nested record types (depth-bounded for self/cyclic types).
Measured: a nested-record read loop (:r (:origin ray)) runs 1.3s with ^Vec3 hints
vs 7.1s without — 5.5x. This is the lever the ray tracer needed (vecs flow through
container fields); records without it read back as :any and stay unproven.
A protocol method call compiles to (protocol-dispatch proto method this rest) — a
runtime registry walk (type-tag -> proto -> method) on every call, ~19x a direct
call. When the inference proves the receiver (arg 0) is a known record type, the
call now resolves to a DIRECT method call at compile time, skipping the registry.
- defprotocol registers each method's var-key 'ns/method' -> [proto method] (a
ctx-capturing register-protocol-methods! emitted into the do-block); infer-unit!
feeds it to the inference via a box (like record-shapes).
- the record-ctor return type carries :type (the record tag) so the inference
knows the receiver type; the :else invoke case annotates a protocol call whose
arg0 has a known :type with :devirt-{type,proto,method}.
- emit-invoke resolves the impl via find-protocol-method at emit time and emits a
direct call to the embedded impl fn value. Unknown/polymorphic receivers (no
proven :type) fall back to the dispatch path unchanged.
Measured: removes the dispatch overhead (14.7s -> 9.3s on a 10M-call loop); the
remaining cost is the method body itself (non-inlined, unproven reads) — inlining
the resolved method is the follow-up (jolt-t6r) toward direct-call speed.
Sound under the closed-world assumption direct-linking already makes (the impl is
resolved + embedded at compile time). Adds devirt-test (subprocess: dispatched ==
devirtualized across polymorphic dispatch, unknown-receiver fallback, and
heterogeneous collections). Stalin's compile-call/callee-environment is the model.
JOLT_WHOLE_PROGRAM (requires direct-linking) defers the per-namespace inference
and runs ONE fixpoint over every user unit at once, so param types propagate
across namespace boundaries — a non-inlined fn's record params get proven from
its callers in another unit, which the per-ns pass can't see. Sound only under
the closed-world assumption (no later eval/redefinition) the flag asserts; slow,
memory-heavy builds are the documented trade-off (the reason it's opt-in).
infer-unit! now takes one ns-name OR a list; infer-program! gathers all recorded
user namespaces and runs the existing fixpoint over the union (re-emit was already
ns-agnostic — keyed by var-key, callee-first). The evaluator defers + records each
unit under the flag; run-main triggers infer-program! after all requires, before
-main. Off by default — per-ns behaviour unchanged.
Measured: a recursive (non-inlined) cross-ns record reader runs 1.66x faster
(8.9s -> 5.3s) — params proven -> bare-index reads. NOTE: small accessor fns are
INLINED cross-ns and records carry GLOBAL declared shapes, so most record reads
are already proven without this pass; the win is for non-inlined hot fns, and it's
the foundation for future whole-program work (devirtualization, unboxing).
Adds whole-program-test (subprocess soundness: per-ns and whole-program produce
identical results on a cross-ns record program).
Records (defrecord/deftype) are now shape-recs in a direct-linking unit by
default — no JOLT_SHAPE flag. A record's shape is DECLARED, so the inference
proves field reads by a lookup, not fragile shape inference, and they bare-index.
Result: ~1.4x faster than the :jolt/deftype table form on a record-heavy loop
(3.9s vs 5.5s), driven by cheaper construction + proven bare-index reads.
Two gates now:
- :shapes? — shape-recs active; records use declared-shape layout + bare
index reads. On with direct-linking (where the inference runs).
- :map-shapes? — also shape generic const-key maps. Opt-in (JOLT_SHAPE), because
shaping maps net-loses on unproven reads (measured). Records win.
- call-ret-type types a record ctor (->Name) as a struct of its declared shape,
fed from a ctx-env registry populated at deftype; field reads on the result
bare-index. (set-record-shapes!/set-map-shapes! wired through infer-unit!.)
- sidx reads the field's position from the :shape vector AS-IS (declared order
for records, str-sorted for map literals) — no re-sort — so any field order
bare-indexes correctly. The map :map case only sets :shape under :map-shapes?.
- record-shape-for interns the descriptor per (type, fields), not per type: a
record redefined with different fields now gets a fresh descriptor instead of a
stale one (fixes redef descriptor staleness; old instances stay valid).
Adds record-declared-shape-test (declared-order reads, incl. non-alphabetical
fields, through fn boundaries + protocol method bodies). Known pre-existing edge
case filed as jolt-wf4 (direct (:f (->R …)) read returns nil after a record is
redefined with different fields; let-bound read works; repros without shapes).
Measurement (vec-heavy benchmark + read-mechanism micro-benchmarks) showed that
shape-rec'ing generic const-key maps is a net loss in a bytecode VM: an unproven
field read can't beat a native Janet struct-get (one opcode), and an inline cache
doesn't transfer to a VM jolt doesn't control. The win is real only for PROVEN
reads (bare-index beats struct) — i.e. construction-heavy or inference-friendly
code, not general map-through-fn code.
So shapes are opt-in again (JOLT_SHAPE), not defaulted on in direct-link. Kept the
get-or-shape fix: the non-proven shape read now indexes inline off the descriptor
instead of calling core-get (which was ~2.5x slower per read). :shapes? is the
single opt-in gate; image cache keys on JOLT_SHAPE + JOLT_NO_SHAPE.
Next: records with declared shapes get fast field access by default (the general
case), which is where the proven-read win actually lands.
Record shape-recs now compare type-aware like the table form: eq-map-pairs
returns nil for a record so equality falls to deep=, which is type-aware via the
per-type interned descriptor. A record equals only a same-type record, never a
plain map (cross-type and record-vs-map were wrongly equal under JOLT_SHAPE).
assoc of a new (undeclared) key keeps the record type and grows a slot, matching
Clojure's record-extension semantics.
Adds test/integration/record-shape-test.janet locking in the runtime record
mechanism (tag, field access, virtual :jolt/deftype, type-preserving assoc,
dissoc demotion, type-aware equality, #ns.Type{...} printing).
Records (defrecord/deftype) become shape-recs whose descriptor also carries
:type, under JOLT_SHAPE (flag off keeps the :jolt/deftype table form). A record
is a Janet tuple [descriptor field0 ...] in declared field order, so the
positional ->Name ctor maps args straight to slots.
- record-tag unifies the type accessor over both reps; instance?/satisfies?/
protocol dispatch and the . interop path go through it.
- virtual :jolt/deftype key on record shape-recs (via shape-get) keeps every
existing (get obj :jolt/deftype) dispatch site working.
- emit-kw-lookup's non-proven fallback routes any tuple to core-get (shape
aware), not gated on compile-time JOLT_SHAPE — core is baked without the flag
but still receives user shape-recs.
- assoc keeps the type (same-shape in place, new key grows a slot Clojure-style);
dissoc of a declared field demotes to a plain map; pr prints #ns.Type{...}.
- JOLT_SHAPE added to the image-cache key (it shapes core's own compiled paths).
The inference dropped the complete :shape whenever it rebuilt a struct type
(cap) or joined two (join-t/merge-fields), so a vec3 retrieved from a container
or a fn param typed across call sites lost its layout and every field read fell
to the slow descriptor path. Two fixes:
- cap preserves :shape: capping truncates field VALUES below the depth limit but
never the key SET, so the layout is still complete. It also recurses into
fields, so a shaped value nested in a container (a vec3 inside a hit-info)
keeps its own :shape — which is what lets (:r (:normal hit-info)) bare-index.
- join-t preserves :shape when both sides are the SAME complete shape (the
merged struct has the same keys); different shapes still drop it. This carries
the shape through if-joins and the inter-procedural fixpoint's call-site joins.
Result: the ray tracer goes from 22s (R1, correct-but-descriptor-path) to 4.36s
— 2.7x FASTER than the 11.7s no-shape baseline, and ~3x the JVM (was 8.5x), with
byte-identical output. The compounding of cheaper tuple construction plus
bare-index reads across the whole render far exceeds the per-op estimate.
Gate green flag-off, suite 4718, default-path bench even, transparency intact.
Removes ALL hardcoding. Every constant-key map literal in user code becomes a
shape-rec (a Janet tuple [descriptor v0 v1 ...]); the descriptor is interned
per key SET with a single canonical key order owned by the runtime
(types/shape-sort), so every site that builds or reads a shape agrees.
Consistency is the foundation: shape-recs are made UNCONDITIONALLY for
const-key user maps (emit-map), not gated on the inference — so a value's
representation always matches what the type system claims, which was the bug
that broke the earlier generalization (a ray built as a struct but bare-indexed
as a shape). Gated on :inline? so it applies to user data only, never to core
or the compiler's own IR-node maps.
Transparency layer (the shape-rec block now lives in types.janet, reachable by
core + evaluator + backend): get, assoc, dissoc, count, contains?, map?, first,
seq, the central realize-for-iteration normalizer (keys/vals/reduce-kv), eq-map-
pairs + eq-seqable (equality), pr-render (print), jolt-call (compiled IFn), and
the interpreter's coll-lookup all handle shape-recs. A new spec
(shape-transparency-test) asserts each op matches the equivalent struct map,
including nil/false values (which shape-recs store positionally — unlike
structs).
The ray tracer renders byte-identically (mean 122.04). Gate green with the flag
off, suite 4718. NOT yet faster — every field read currently takes the
descriptor path because the inference drops the complete :shape through joins
and containers; that's Round 2 (completeness preservation). All behind
JOLT_SHAPE (off by default).
Removes the {:r :g :b} hardcoding. ANY constant key set is now a shape:
- inference: a struct type from a map LITERAL carries :shape (its canonical
str-sorted key vector — completeness); joins/access-inferred structs lack
it, so they never get a bare index. The literal node and lookup subjects
carry the shape; the back end derives the index from it.
- backend: emit-map turns any shape-tagged const-key map into a shape tuple;
emit-kw-lookup reads the field by bare index when the complete shape is
proven, else by the value's own descriptor (so a shape-rec whose :shape was
dropped by a join still reads correctly).
- runtime: core-get and core-assoc handle shape-recs.
Status: CORRECT for direct field access, container round-trips, and assoc
(minimal repros pass). NOT yet complete — the full ray tracer still hits an
uncovered path (a shape-rec reaching a map op without coverage: keys/vals/
count/seq/equality/print/jolt-call/dissoc/contains?/the interpreter's
coll-lookup all still need shape-rec branches). And the perf win needs
COMPLETENESS PRESERVATION through joins/containers (merge-fields/cap drop
:shape today, so nested vec3 access falls to the descriptor path, slower than
a struct get) — without it the general version is slower than the vec3
prototype.
All behind JOLT_SHAPE (off by default). Gate green with the flag off, suite
4718. This preserves the general design; the transparency layer + completeness
preservation are the remaining multi-session work.
Validated prototype of the hidden-class object-model change. A vec3-shaped
{:r :g :b} map literal is represented as a cheap Janet tuple [shape vb vg vr]
instead of a struct (~2x cheaper to construct); a lookup on a value the
inference PROVES is the shape reads by bare index with no runtime check.
Result on the ray tracer (direct-link): 12.3s -> 10.7s (~13% faster), with
byte-identical pixel output. The shape value flows transparently through
hit-info/ray/material containers and the colors vector; core-get handles it
(inline check, no fn call) so an unspecialized access is still correct.
Key lessons baked in: the lookup MUST compile to a bare index (a runtime
shape check, even inlined, taxed every field read and made it 2.5-3.4x
SLOWER) — so the inference gained a :shape hint (struct type with keys
exactly {:r :g :b}) that the back end turns into (in m idx). The descriptor
is quoted when embedded (its keys are a parens tuple Janet would otherwise
try to CALL).
All behind JOLT_SHAPE (off by default). Gate green, suite 4718, default-path
bench even. Scoped to the one shape; NOT yet sound in general (assumes every
vec3-shaped value is a shape-rec, true under the flag for the ray tracer) nor
fully transparent (only core-get + the inlined lookup; jolt-call/equality/
print/keys not yet covered). Those are the next steps toward a real feature.
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
Two try bugs that blocked migratus's migrate (run/table-exists? are multi-body
try/catch/finally with janet-interop jdbc calls, which punt to the interpreter):
- The interpreter's try took only form 1 as the body, dropping later body forms
before the clauses, and did not run finally on the success path when a catch
was present. Rewritten to collect every body form before the first
catch/finally and to always run finally (via defer, so it fires even if a
catch body throws).
- current-ns leaked after a caught throw from an INTERPRETED fn (it sets ns to
its defining ns and can't restore on unwind). The interpreter's try now
restores; the compiled try (emit-try) snapshots the ns at entry and resets it
in the catch via new __current-ns/__set-current-ns! core fns. Statement
values also get a :close key so with-open can close them.
With these, migratus migrate/rollback round-trips on jolt (verified end to end
against sqlite). Conformance 335/335 x3, full gate green.
methods/get-method now take the multimethod VALUE (Clojure semantics), so the
arg must resolve to compile. The isolated analyzer never defines mf, so point
these two at print-method (a real defmulti) instead.
File API (jolt-hjw): io/file and (File. …) build a tagged :jolt/file value
(instance? File true) with a full method surface (isFile/isDirectory/exists/
getName/getPath/getAbsolutePath/listFiles/toPath/delete/createNewFile/…) backed
by os/ and file/. file-seq is File-aware (leaves are File values). str/slurp/spit
coerce :jolt/file to its path. ClassLoader/getSystemClassLoader + a classloader
stub whose getResource returns nil degrade migratus's classpath lookup to the
filesystem. java.nio.file Path/FileSystem/PathMatcher are shimmed just enough for
script-excluded?'s glob (recursive * / ? matcher).
Three bugs found getting migratus's migration discovery to work:
- (assoc nil k v) returned a raw janet table, not a map, so assoc-in built tables
that count/seq rejected. Now returns an immutable map.
- methods/get-method resolved the multimethod symbol at runtime in the current
ns, so a bare multifn ref in its defining ns saw an empty table once defmethods
lived elsewhere. Now they take the multimethod VALUE and recover the var via a
registry (Clojure semantics).
- defmulti now drops a leading docstring/attr-map (migratus's multimethods carry
docstrings) instead of treating the docstring as the dispatch fn.
Conformance 335/335 x3, clojure-test-suite at baseline.
The analyzer always took (nth items 2) as the value, so (def x "doc" 42)
bound x to the docstring and dropped 42. Now it mirrors the interpreter:
when there are 4+ items and item 2 is a string, item 2 is the docstring
(attached as :doc meta) and item 3 is the value. Conformance 335/335 x3.
For the migratus next.jdbc shim (jolt-0z5):
- core.janet: __jdbc-wrap-conn / __jdbc-conn-raw / __jdbc-make-stmt builtins.
A connection is a tagged wrapper over a jdbc.core conn carrying a clj :exec
callback so the host Statement.executeBatch runs SQL without a janet->clj call.
- javatime.janet: tagged-methods for :jolt/jdbc-conn (setAutoCommit/isClosed/
close/getMetaData), :jolt/jdbc-meta (getDatabaseProductName), :jolt/jdbc-stmt
(addBatch/executeBatch/close); java.sql.Timestamp ctor -> millis.
- evaluator.janet: instance? case for Connection/java.sql.Connection so
migratus's do-commands runs SQL through its Connection branch.
Two defn/defmacro fixes found loading migratus.core (both rooted in the reader
representing ^{:map} name metadata as a with-meta form, jolt-8w2):
- defmacro special form: unwrap a with-meta name (mirrors def), and handle the
arity-clause form (defmacro name ([params] body...)) like fn/defn — a params
vector reads as a tuple, an arity clause as a list (array).
- defn overlay: pass the bare (unwrapped) name to fn while def keeps the meta.
Conformance 335/335 x3 modes.
Faithful shim of the real clojure/tools.logging public API rather than a
bespoke logger. clojure.tools.logging.impl provides the Logger/LoggerFactory
protocols (matching upstream signatures) plus a jolt stderr-backed factory;
find-factory returns it instead of probing slf4j/log4j/jul. The macro surface
(logp/logf, level macros + f-variants, log, enabled?, spy) dispatches through
those protocols. Departures forced by no-JVM: log* writes directly (no agent/
LockingTransaction), and the throwable-first-arg branch is dropped since
(instance? Throwable x) is always false for jolt exception values.
Workarounds for two jolt gaps found en route (filed as jolt-9av, jolt-6ym):
macro bodies fully-qualify impl refs because syntax-quote does not resolve ns
aliases, and def docstrings are kept in comments because (def name doc value)
is mishandled.
stderr via __eprint. trace/debug suppressed by default (impl/*level* :info).
Conformance 335/335 x3 modes.
Two provably-wrong cases the inference already has the facts for, closing the
last RFC 0006 open question:
- Calling a non-function. At an :invoke whose callee is provably :num or :str
(the only non-callable types — keywords/maps/vectors/sets are IFn), report
"cannot call a number as a function". Default level (no closed-world: the
callee type is inferred at the call site). Covers (5 1), ("hi" 0),
((+ 1 2) :k), a let-bound number, and a var holding a number (via vtype-box
in direct-link). A union is non-callable only when every member is, so
((if c 1 :k) x) is accepted (:kw is callable). Verified zero false positives
on the ray tracer, which calls maps/keywords/vectors as fns throughout.
- Wrong arity to a user fn. The registered single-fixed-arity sig (jolt-zo1)
makes a mismatched arg count provably throw; reported under the
JOLT_TYPE_CHECK_USER opt-in (same closed-world boundary; ^:redef/variadic
skipped). Caught at compile time before the runtime arity error.
Both fold into the existing infer walk, carry :pos for file:line:col, and keep
no-false-positives. Gate green, suite 4718, conformance 335/335, runtime bench
even (compile-time only).
Character/isUpperCase + isLowerCase (ASCII, on :jolt/char structs);
Thread/interrupted (false, no real threads) + Thread/currentThread stub
with a .getContextClassLoader classloader stub; Long/valueOf; java.net.URI
constructor (string round-trip); and a minimal java.util.Date / TimeZone /
SimpleDateFormat supporting the y M d H m s tokens migratus's timestamp uses,
backed by os/date (UTC default). Full gate + 3-mode conformance green.
jolt-6xk: resolve bare exception class symbols (Exception,
IllegalArgumentException, InterruptedException, Throwable) by consulting
class-ctors/class-canonical-names in the unqualified symbol-resolution
fallthrough, mirroring the qualified path. Constructors already existed
in javatime; throw/catch/.getMessage already handled the string payload.
jolt-47b: java.util.regex.Pattern statics (compile/quote/MULTILINE) that
return jolt's native :jolt/regex value so str/replace, re-matches, and
.split accept them transparently. MULTILINE maps to a (?m) prefix routed
through the regex engine's inline-flag path; the engine gains (?m) anchor
support with the non-multiline branches left byte-identical. String
methods .matches/.replaceAll/.replaceFirst added. .split dispatches on
compiled-regex via a :jolt/regex tagged-method.
Full jpm test gate green.
Update the status, strictness levels, and open questions to reflect what
landed: bounded unions (jolt-pz5), user-fn domains behind
JOLT_TYPE_CHECK_USER (jolt-zo1), precise file:line:col (jolt-fqy), and the
checker folded into one inference walk that piggybacks on direct-link
specialization (on by default there, opt-in in plain builds). Align the
error-reporting example with the actual output format.
Checking inherently needs an inference pass (~2.6x compile as a standalone
pass). But direct-link builds ALREADY run one inference pass for
specialization (run-passes' infer-top), so checking can ride along: set a
check-mode flag, turn checking? on during that existing pass, and collect
the diagnostics after — ~2% overhead measured on the ray tracer, vs 2.6x
for the separate pass.
So the checker now defaults to `warn` in direct-link builds (where it's
nearly free) and stays OFF in plain REPL/dev builds (no inference to ride,
no forced cost — opt in with JOLT_TYPE_CHECK there). JOLT_TYPE_CHECK still
overrides in both directions (off to disable, error to escalate).
It checks the POST-optimization IR, which matches what the optimized
program actually evaluates — scalar-replace only drops provably-pure code,
an accepted opt-mode divergence, so no real error is hidden. The loaders
enable position tracking whenever checking will run (env-selected or
direct-link). type-check! (the standalone pass) stays for plain builds;
both paths share report-diags!.
cli-test pins: plain build silent, direct-link warns by default,
JOLT_TYPE_CHECK=off disables. Gate green, suite 4718, runtime bench even.
The checker ran a separate check-walk that re-inferred each argument's
subtree AND recursed into it — quadratic in expression nesting. Fold the
diagnostic emission into `infer` itself (gated by a checking? flag, off
during the optimization fixpoint): one O(n) walk that both types and
checks. Removes check-walk entirely; check-form now drives infer.
This is a cleanup and removes the deep-nesting blowup, but it does NOT make
warn-by-default cheap: on a real 360-line file the checker still adds ~2.6x
compile time (277ms -> 720ms). That cost is the structural inference pass
itself, which checking inherently requires — not redundancy. A cheap
default-on path would need either piggybacking on the inference direct-link
already runs, or a lighter scalar-only checker inference. Gate green,
type-check tests pass.
rewrite-message assumed janet's BINARY arithmetic dispatch error shape
("could not find method :+ for 1 or :r+ for "a""). Unary inc/dec/- on a
non-number produce "could not find method :+ for "x"" — no "or :r" clause
— so orpos was nil and the reporter itself threw "could not find method :+
for nil", burying the real error. Handle the unary form. Found auditing
the RFC 0006 checker's default (checker-off) path. Regression row in
cli-test.
RFC 0006 error reporting wanted file:line:col but IR nodes carried no
position, so diagnostics read only "type error in <ns>: <msg>". Now:
type error /tmp/scene.clj:5:5: `inc` requires a number, but argument 1 is a string
The reader records each LIST form's absolute start offset in a table keyed
by form identity (lists are fresh arrays, never interned), gated behind a
flag the loaders enable only when JOLT_TYPE_CHECK is on — zero cost off.
Keying by identity makes positions survive macroexpansion exactly when the
user's own sub-form is spliced through, and absent for macro-synthesized
structure: a `(inc :k)` written inside `(when c ...)` reports at its own
line, never at the expansion's generated if/do.
The analyzer stamps the offset onto :invoke nodes (form-position host
contract fn); the checker carries it into each diagnostic as :pos; the
loaders stash the file's source + path on the env (save/restored across
nested requires); backend/type-check! converts offset -> line:col via the
reader's line-col and renders the RFC format. Falls back to the ns when no
position is available (synthetic forms), so it is never worse than before.
Gate green, conformance 335/335, suite 4718, runtime bench even (positions
are compile-time only; off by default).
The success checker fired only against core-fn error domains (stable, not
redefinable). This adds reporting of a call that passes a provably-wrong
type to a USER fn whose body requires otherwise — e.g. a fn that only does
arithmetic on a param, called with a string.
As check-walk sees defs it registers each non-redefinable single-fixed-arity
user fn's {:params :body} in module state (user-sig-box, accumulating across
forms like rtenv-box — a def must precede its call). At a call site (strict
mode only) the body is re-checked with ONE parameter bound to its concrete
argument type, others :any; if that produces a diagnostic the all-:any body
did not, the argument alone is provably wrong and the call is reported.
Monotonic — binding a concrete type can only add error-domain hits — so still
no false positives. A cycle guard (checking-box) terminates mutual recursion.
Gated behind JOLT_TYPE_CHECK_USER (orthogonal to the warn/error level)
because it rests on the closed-world assumption, weaker than the core-fn
case. check-form gains a strict? arity; the default path is unchanged and
user-fn code runs only when the checker is enabled. ^:redef/^:dynamic and
multi/variadic fns are not registered (their body is no stable requirement).
Gate green, suite 4718, conformance 335/335.
The success checker (RFC 0006) used to lose differing if-branches to :any
and accept the use. (inc (if c "a" :k)) typed the if as :any — sound but
imprecise, since the value is provably {:str | :kw}, every member of which
is in inc's error domain.
Adds {:union #{T...}} to the lattice: join-t forms a scalar union of
differing branches instead of collapsing to :any, capped at 4 distinct
scalars (the member space is the five scalar tags, so the lattice stays
finite and the inter-procedural fixpoint still terminates). The checker's
not-number?/not-seqable? report a union only when EVERY member is in the
error domain — any valid member accepts the call, so still no false
positives. type-name renders "a string or a keyword".
Unions are scalar-only and carry no :struct/:vec/:set key, so every
structural predicate already treats them as opaque — specialization sees
them exactly as :any and codegen is unchanged. Gate green, suite 4718,
conformance 335/335, bench even.
Reuse the structural inference from RFC 0005 as a loose type checker. It reports
a core-fn call only when an argument's inferred type is concrete and lies in
that op's throwing error domain, and accepts everything ambiguous (:any, a
union that joined to :any, :truthy). By construction it never produces a false
positive: a correct program has nothing to report even in error mode.
The curated error-domain table starts with the clearest throwing cases:
arithmetic on a provable non-number, and count/first/rest/next/seq/nth on a
provable non-seqable scalar. Lenient operations like (get 5 :k) and (:k 5),
which return nil rather than throw, are deliberately not listed.
Checking is decoupled from specialization: it runs whenever JOLT_TYPE_CHECK is
warn or error, regardless of :inline?, reading the knob at compile time so no
rebuild is needed. warn prints to stderr, error fails the form's compilation,
off (the default) skips it entirely. Core init stays clean under the flag.
jolt-y3b
Replace the ad-hoc inference lattice (a flat :struct-map tag plus {:vec ELEM})
with one recursive structural type: {:struct {field -> T}}, {:vec T}, {:set T},
scalar tags, and :any. A keyword lookup now returns its field's type, so nested
access like (:r (:direction ray)) is typed end to end and drops its guard. join
is field-wise and element-wise with a depth cap of 4 so the inter-procedural
fixpoint still terminates.
The back end honors a struct hint on any subject node, not just locals, so an
inferred field type on a nested lookup specializes. The orchestrator's fixpoint
joins through the portable join-types so compound types no longer collapse to
:any.
Ray tracer goes 12.8s to 11.0s with no hints, matching the explicit ^:struct
version (10.9s). Render checksum unchanged (1915337), full gate green,
conformance x3 modes pass.
jolt-5uj
0005 proposes replacing the ad-hoc inference lattice with one recursive
structural type (a struct carries its field types, a vector its element type,
recursively), so a lookup returns its field's type and nested access is typed
end to end. It unifies :struct tracking with field tracking, subsumes the
current inference phases, and is the soft-typing (HM + a dynamic top) design:
structural types + core-fn type schemes, solved by lattice join with :any as
top instead of unify-or-fail. Includes the depth cap for termination and an
explicit design-problems section.
0006 (follow-up, depends on 0005) reuses the inference as a loose type checker
in the success-typing discipline (Dialyzer): report only PROVABLY-wrong code
(a concrete type in an operation's throwing error-domain), accept everything
ambiguous, never a false positive. Curated error-domain table, strictness
levels (off/warn/error), clear located messages, and the soundness boundaries
(closed-world, macros, unions).
The inference now tags a :local it proved to be a vector with :hint :vector, and
the back end specializes (count v) -> pv-count (skipping core-count's dispatch
chain) and the 3-arg (nth v i default) -> pv-nth. The 2-arg nth is deliberately
NOT specialized: pv-nth returns nil out-of-bounds where Clojure nth throws.
Sound, conformance 335/335 x3 and full jpm test pass; type-infer-phase2-test
pins the specialization and the 2-arg exclusion.
Extends the inference lattice with a parametric vector type {:vec ELEM} and
threads element types through the program:
- vector literals, conj/into, and range produce element-typed vectors;
- reduce/map/mapv/filter/filterv seed their closure's element (and reduce's
accumulator) param, so a lookup inside the closure over a vector-of-structs
specializes (the HOF-element-awareness piece);
- a var reference carries a VALUE type — a fn var is :truthy (non-nil, sealed
root), a def var carries its inferred init type (e.g. a color table is
{:vec :struct-map}); element-returning fns (rand-nth/first/nth/...) yield the
collection's element type. These let the dynamically-built scene's sphere
maps type as structs.
The inter-procedural fixpoint now also infers non-fn def value types, and the
recompile re-emits the WHOLE unit callee-first (reverse-topological) so a
caller re-embeds its recompiled, now-specialized callees and a call site
compiled after the pass links the whole chain.
Result on the ray tracer (no hints): the chain closes — hittables infers to
{:vec :struct-map}, hit-sphere's hittable param to :struct-map — and the render
goes 13.1s -> 12.8s. That is only ~3%, far short of the explicit hint's 1.22x.
The remaining gap is nested field access: a lookup RESULT like (:direction ray)
is :any, so (:r (:direction ray)) stays guarded, and the vec3 fns (called with
such values) can't be typed struct. The hint asserts the vec3 params directly
and propagates through inlining; matching it needs field-shape types
(ray.direction : vec3, vec3.r : number) — a structural extension (Phase 4).
Sound: a seeded full render produces an identical checksum (1915337);
conformance 335/335 x3 and the full jpm test pass; type-infer-phase3-test pins
the element-typing + HOF mechanism. Phase 2 (vector nth/count specialization)
was deprioritized — it is orthogonal to this benchmark.
Closed-world (optimization mode): after a unit loads, infer-unit! runs a
whole-unit fixpoint over the call graph and recompiles. A fn's param types are
the lub of its in-unit call-site arg types; its return type is the lub of its
tail positions; iterated to a least fixpoint. Param types are RECOMPUTED FRESH
each iteration (not accumulated) because :any is the lattice top — joining an
early-iteration :any would poison the result permanently. Closures inherit the
enclosing tenv so captured locals keep their types (their own params shadow to
:any). A fn whose var escapes as a VALUE keeps :any params (its callers aren't
all visible). Each fn is then re-inferred with its param types seeded and
re-emitted; recompiled bodies are semantically identical, so correctness holds
regardless of order. Sound under source distribution + whole-program compile
(the consumer compiles all call sites together).
Plumbing: the portable pass (jolt.passes) gained inter-procedural primitives —
set-rtenv!, infer-body (types a body, collects its call sites), reinfer-def
(seeds param types), and escape tracking. The back end stashes each
single-fixed-arity defn's :def IR (:infer-ir); the evaluator triggers
infer-unit! after a unit loads (via an env hook, opt mode only).
Result and honest finding: the fixpoint correctly types scalar-flowing params
(ray-cast/hit-all/hit-sphere all get the ray param as :struct-map, no hint),
but the ray tracer does NOT speed up — its dominant lookups are on `hittable`,
the element of the `hittables` vector threaded through `reduce`, which stays
:any. Typing it needs collection-element types (vector<struct>) plus HOF-element
awareness (knowing reduce applies the closure to elements), which is beyond
inter-procedural param inference. The explicit ^:struct hint reaches it (it
types the reduce closure param directly), which is why the hinted run is 1.22x.
Verified: conformance 335/335 x3, full jpm test; new type-infer-phase1-test
pins the fixpoint, the escape gate, the seeded re-inference, and correctness.
A forward, soft-typing-style pass (simplified HM: monovariant, never-fails,
lattice top = :any) in jolt.passes, run after the inline/scalar-replace
fixpoint when the optimization mode is on. It types expressions from literals
and arithmetic, flows the type through let bindings, and joins at if-branches.
Where a keyword-lookup subject is PROVEN to be a plain struct map it sets
:hint :struct (the same channel a manual hint uses, so the back end drops the
:jolt/type guard); where the type is :any it leaves the dynamic guard in place.
Sound by construction: a concrete type is assigned only when proven (scalar
keys with non-nil/non-false values for a struct-map), so a wrong bare get can't
happen. This is the foundation; on its own it mostly overlaps Route 1
scalar-replacement (which already eliminates non-escaping let-bound maps), so
its standalone win is small. Phase 1 (inter-procedural) is where escaping
params get typed.
Verified: conformance 335/335 x3, full jpm test; new type-infer-test pins the
flow rules and the sound :any fallback (cases force the map to escape so the
test isolates inference from scalar-replacement).
Builds on the ^:struct keyword-lookup hint:
- ^TypeName for records. A tag naming a defrecord/deftype now resolves to the
struct fast path: record instances are tables tagged :jolt/deftype (not
:jolt/type), so a raw keyword get is correct for them. A new host contract fn
record-type? detects a record by its ->Name constructor; a non-record tag
(^String, ^long, ...) is ignored, as before.
- (get m :k) and (get m :k default) now get the same inlined keyword lookup as
(:k m): the representation guard fast path when unhinted, and the bare get
when the subject is ^:struct/^Record. A variable/number/string key still
falls through to core-get. The two call shapes share one emitter
(emit-kw-lookup).
- JOLT_CHECK_HINTS=1 turns a violated hint into a clear runtime error (naming
the local and key) by keeping the guard and throwing on the tagged arm. It is
off by default with zero cost to normal builds (a hinted lookup still emits a
bare get), and is part of the image-cache fingerprint. This is the answer to
"a lying hint is silent": opt into checking during development.
- Docs: RFC 0004 records the design, soundness contract, and measurements; the
reader spec gains S12b (hints are semantically transparent; jolt recognizes
^:struct and ^Record as lookup-optimization assertions).
There is no Clojure keyword equivalent for "plain map / fast keyword access"
(Clojure hints are class names), so ^:struct stays a jolt-specific flag,
analogous to ^:dynamic.
Verified: conformance 335/335 in all three modes and the full jpm test pass; a
seeded ray-tracer render is byte-identical hinted vs unhinted; the struct-hint
test covers record hints, the get-form, inline propagation, and the checked-mode
error. Full render with hints holds at 13.3s -> 10.9s (1.22x).
A constant-keyword lookup (:k m) currently emits a guarded form,
(if (get m :jolt/type) (core-get m k) (get m k)), to tell a plain struct
(raw get is correct) from a phm/sorted/transient (needs core-get). On a
struct that guard is a second get, so the lookup costs ~36ns where a bare
get is ~20ns. Profiling the ray tracer (jolt-dad) showed keyword lookups are
~50% of a render and the guard is the only avoidable part, but dropping it
needs to know statically that the subject is a plain struct.
Type hints are exactly that information, and jolt already parses them and
otherwise ignores them. This wires one through: a local hinted ^:struct
asserts a plain struct/record map, so a (:k local) lookup on it skips the
guard and emits a bare get. The hint rides on the binding symbol into the
analyzer, which records it per-local and attaches it to :local IR nodes; the
back end reads it on the lookup subject. It also propagates through inlining:
when the inliner let-binds a non-trivial arg to a fresh local, it carries the
called fn's param hint onto that local, so lookups inside the spliced body
keep the bare path. This is a programmer assertion, like a Clojure type hint
(an inaccurate hint just makes the raw get return the wrong value, the same
contract as a wrong ^String), so it stays opt-in and off by default.
On the ray tracer (with inlining on) this is 13.3s to 10.9s, 1.22x, taking it
to 7.8x JVM from 9.4x after the inline pass. The unhinted path emits identical
code (the fast arm is just factored out), so nothing changes without hints.
Verified: a seeded full render produces an identical checksum hinted vs
unhinted; conformance 335/335 in all three modes and the full jpm test pass;
new test/integration/struct-hint-test.janet pins the guard removal, the
inline propagation, and that an accurate hint is correctness-preserving.
Adds two IR passes to jolt.passes that run when a unit opts into
direct-linking (JOLT_DIRECT_LINK=1, off by default). The inline pass splices
small direct-linked fns at their call sites, copy-propagating trivial args so
that scalar replacement can then see map literals across the call boundary.
Scalar replacement is AOT escape analysis: a map allocation whose only use is
constant-keyword lookup is dropped and each (:k m) is replaced with the value
at :k, both for a literal lookup subject and for a non-escaping let-bound map.
Inlining and scalar replacement iterate to a capped fixpoint, since inlining
exposes literals that scalar replacement then collapses.
The back end stashes the body IR of each single-fixed-arity defn on its var
cell (inline-stash!), and the portable pass reads it through two new jolt.host
contract fns (inline-enabled?, inline-ir). Inlining is gated on :inline?, which
is off for all of init so core and the self-hosted compiler compile exactly as
before (const-fold only); api/init and main re-read JOLT_DIRECT_LINK so the
flag works both for a freshly built context and for the build-time-baked one in
the shipped binary.
Only inline-safe targets are spliced: a single fixed arity, no recur/loop/fn/
try crossing the boundary, within a size budget, a closed body (no free locals
beyond the params, so a self-recursive fn's name reference can't dangle), and
not ^:redef / ^:dynamic. Bodies are fully alpha-renamed so no spliced name can
collide with a caller local.
On the ray tracer this is 15.3s -> 13.0s (1.18x). The ceiling is honest: that
workload's cost is dominated by lookups on maps that genuinely escape (rays,
hits, materials) and by dynamic dispatch (the reduce closure, the :scatter fn),
which escape analysis cannot remove. On allocation-bound code where the
temporaries are local it is far larger: a vec3 reflect+dot loop goes 9.3s ->
0.38s (25x), with the loop body reduced to pure arithmetic.
Verified: full jpm test passes (inline off, no regression); conformance 335/335
in all three modes and the clojure-test-suite both pass with inline on; new
inline-sra-test pins the transform and its semantics.
jolt-p3c: Clojure evaluates map-literal entries left to right, but the
reader represented map forms as bare janet structs, so entries ran in
hash order. The reader now carries [k v ...] source order out-of-band —
on a struct PROTOTYPE (keys/kvs/length ignore protos, so macros that
get/keys literal map forms see no change; jolt-equal? was already
structural) and as a plain field on the phm rep (nil key/value). The
analyzer (form-map-pairs), the interpreter's map eval, both
syntax-quote walks, and core-sqmap (the lowered `{...} builder — the
array-map case, where Clojure also preserves insertion order) all honor
it, so the order survives macroexpansion in both modes.
jolt-507 root-caused: the parked inline put a LOCAL in janet call-head
position for the first time, and janet resolves head symbols against
the macro table before lexical upvalues — clojure.core/repeat's
self-name local expanded as janet's (repeat n & body) macro, compiling
the self-call into a countdown loop returning nil. Everything in the
issue (interpose, interleave) traced to that one name collision. The
emitter now rebinds local callees to reserved _fp$ symbols (argument
positions never consult the macro table), and the inline — direct
calls for function locals, jolt-call only for IFn-collection
leftovers — lands. Spec rows pin locals named repeat/seq/with called
in head position.
Gate green, suite 4718 steady, bench even with main.
jank's ray tracer benchmark (examples/ray-tracer) drops from 165.6s to
15.8s per render (10.5x) — from 118x JVM Clojure to 11x. The changes
mirror the optimizations in jank's June 2026 post, adapted to the
janet backend (jolt-4vr, jolt-h79):
- (:kw m) emits an inline lookup instead of variadic jolt-call ->
core-get's predicate chain. The guard is (get m :jolt/type): janet
compiles get to an opcode (~17ns) where a struct? cfunction call
costs ~85ns/lookup. :jolt/type is reserved (the reader rejects it in
map literals) and every table rep that must not be raw-indexed
carries it — phm tables now tagged too — so tagged values route to
core-get and everything else gets janet get, which matches core-get
for keyword keys on structs/records/nil/arrays. 929ns -> 90ns.
- {:k v ...} literals with scalar const keys emit let-bound values, an
`and` truthiness test (pure branch opcodes), and a native (struct ...)
call instead of variadic build-map-literal + runtime kv re-scan. nil
or false values fall back to build-map-literal, which keeps Clojure's
nil-entry semantics via the phm rep. 890ns -> 246ns.
- native-op additions: min/max (janet's are variadic with the same
numeric semantics), nil?/some? lowered to janet's fastfun = / not=
against nil, and not.
- clojure.math (Clojure 1.11) installed as a namespace whose vars hold
janet's math natives directly, so calls direct-link. Math/sqrt-style
interop stays in the frozen interpret-only punt set (~5us/call); this
is the compiled route (~30ns).
- reduce over pvec/tuple/array iterates indexed in place — it was
copying the whole pvec into a fresh array on every reduce call — and
stops at `reduced` instead of scanning the tail.
- interpreter coll-lookup gains the sorted-coll arm: (:k (sorted-map ..))
was nil in interpret mode (compiled mode had it right).
map-fastpath-spec pins keyword-invoke/map-literal semantics (16+15
rows incl. nil-value maps, records, sorted, vectors-of-internals) and
clojure.math (10 rows). Two pre-existing bugs found and filed while
writing it: map literals evaluate entries in reader-hash order
(jolt-p3c), and an attempted local-callee call inline that breaks
overlay lazy self-recursion is parked with a repro (jolt-507).
Gate green, conformance 335/335 x3, suite 4718 steady, core bench even
with main back-to-back.
Syntax errors were positionless ('Unterminated list', no idea where).
Now they use Clojure's shape:
Error: Syntax error reading source at (src/app/syn.clj:3:8): Unmatched delimiter: ]
at src/app/top.clj:1
The reader's 15 error sites raise a {:jolt/reader-error :msg :pos}
struct carrying the byte offset (every site already had pos in scope).
The parse entry points convert offset -> line:col on demand and
re-raise the formatted message: parse-string against its own string
(no file), parse-all-positioned against the full source with the
file threaded in from the loaders — rebasing slice-relative offsets
onto the original source so positions stay absolute. No per-token
cost; nothing is tracked until an error actually happens.
Unmatched-delimiter messages match Clojure's 'Unmatched delimiter: )'
wording. cli-test rows assert positions for unterminated string/list,
unmatched delimiter through a require (composing with round 4's
'while loading' chain), and bad ## tokens. Gate green, suite 4718
steady, bench within noise.
A failing top-level form now reports where it lives:
Error: Cannot add 1 and "boom" — + expects numbers
at /path/src/app/broken.clj:3
while loading /path/src/app/mid.clj
while loading /path/src/app/top.clj
The reader has no per-form positions (round 5), but the loaders know
exactly which slice of source each form came from: parse-all-positioned
(reader) counts newlines around parse-next and returns [form line]
pairs; load-ns, load-string and load-ns-source evaluate through a
positioned loop that on error stashes the innermost form's {:file
:line} on the env and appends each file unwound through to a loading
chain. report-error prints both, suppressing the synthetic <eval>
strings the CLI feeds itself (the require/apply one-liners).
load-string takes an optional file arg; run-file passes the script
path so script errors name the script. cli-test rows cover the
3-requires-deep case, script files, and that one-line -e output stays
clean. Gate green, suite 4718 steady, bench within noise.
A typo'd symbol used to auto-intern an unbound var and die later as
'Cannot call nil as a function' with no hint which symbol. Now:
$ jolt -e '(undefined-fn 1)'
Error: Unable to resolve symbol: undefined-fn in this context
The analyzer's :unresolved fallthrough now punts to the interpreter
(whose resolver raises the message above when the form runs) instead of
emitting a var-ref that interned the var. A punt rather than a hard
throw because runtime-interning forms (defmulti's setup) legitimately
reference the var they're about to create from a nested do.
Pulling that thread surfaced three real bugs the leniency was masking:
- h-resolve-global resolved unqualified symbols against ctx-current-ns,
which during analysis is jolt.analyzer — so user-ns vars NEVER
resolved through it; the lenient arm happened to emit the right ns.
Now resolves against the compile ns like the qualified branch.
- Top-level (do ...) wasn't split: Clojure compiles and EVALS each
child in sequence so earlier children's runtime effects (defmulti's
intern) are visible while later children compile. eval-toplevel now
splits.
- The stdlib itself had forward references the auto-intern hid:
10-seq's transducers used vreset!/vswap! from 20-coll (moved to
10-seq); in 20-coll qualified-ident?/realized?/list*/underive
referenced defs declared later in the file (reordered); sorted? and
partition-all are genuinely later-tier and got (declare ...).
Test rows updated where they encoded the old leniency: ir-passes'
dead-branch row (unresolved in a dead branch is an error, as in
Clojure), compile-mode's ctx-isolation row (other ctx now errors
instead of reading nil), cli rows assert the new message. Gate green,
conformance 335/335 x3, suite 4718 steady, bench within noise.
Before: (+ 1 "a") printed 'could not find method :+ for 1 or :r+ for "a"'
followed by three janet frames pointing at jolt internals. After:
Error: Cannot add 1 and "a" — + expects numbers
at app.deep/level3
Round 1 — compiled fns carry their Clojure identity:
- The analyzer's recur target (which doubles as the compiled janet fn's
name) is now ns/fn-name (_r$app.deep/level3--N), so janet stack traces
name the user's fns; defn passes the self-name through to fn.
- eval-toplevel re-raises with propagate instead of protect+error — the
failing fiber's stack was being discarded, which is why every trace began
at eval-toplevel.
- require/maybe-require-ns route loaded namespaces through the loader's
compile-or-interpret eval-toplevel via a ctx hook (the evaluator can't
import the loader). Previously REQUIRED namespaces always ran interpreted:
slower, and their fns were anonymous in traces.
Round 2 — report-error presents for users (rephrase-inspired):
- The full trace text is stashed at the innermost eval-toplevel boundary
(janet's debug/stacktrace walks the fiber propagation chain; debug/stack
cannot), then filtered: _r$ frames demangled to ns/fn-name, jolt-internal
and [eval] frames dropped. JOLT_DEBUG=1 restores the raw janet trace.
- Message rewrites: janet arithmetic dispatch -> 'Cannot add X and Y — +
expects numbers'; compiled arity -> Clojure's 'Wrong number of args (N)
passed to: ns/fn'; nil-call gets an undefined-symbol hint (round 3 will
fix resolution properly).
6 cli-test rows assert the exact user-visible output. Gate green, suite
4718 steady, bench within noise.
Follow-ups for running reitit alongside default-feature libraries in one app:
- __reader-features / __reader-features-set! exposed to Clojure so a namespace
can load a clj-targeted lib (reitit, under :clj) WITHOUT forcing the whole
process to :clj — set features, require, restore. Honeysql/selmer/ring stay
on the default set they were validated under. (jolt vector args coerced to a
janet array — janet map over a pvec iterates keys otherwise.)
- clojure.template added to the stdlib (verbatim, pure Clojure over
clojure.walk) — honeysql's :clj branch requires it.
- java.util.Locale registered as a qualified alias (+ ROOT) for selmer's :clj
Locale/US use.
Everything reitit-core needs to load unmodified from git under :clj features:
- The baked binary now re-reads JOLT_FEATURES at startup (like JOLT_PATH).
reader-features-set! runs at module load = BUILD time for a binary, so a
process opting into :clj (to read a lib's :clj branches) was ignored, and
unmatched #?(...) forms silently spliced to nothing — defn of a fn with an
empty arglist, hence the cryptic index errors.
- (get s i) indexes a string and returns the char, as in Clojure (nth did;
get returned nil). reitit's path parser is (get path i)-based — without
this every route read as static.
- Class-shim registration exposed to Clojure: __register-class-statics! /
__register-class-methods! / __register-class-ctor!, so a library can mirror
a Java class jolt doesn't ship (the reitit.Trie mirror lives in jolt-lang/
router on top of these).
- Java surface reitit's :clj branches call: .getMessage (on exceptions and
strings) and a small universal object-method set, .intern, java.util.HashMap
(a mutable map wrapper). Plus defprotocol already took keyword options.
Gate green; clojure-test-suite 4715 -> 4718 (the get-on-strings fix).
Clojure's defprotocol takes an optional docstring and leading keyword
options (:extend-via-metadata true) before the signatures; jolt's macro
fed the option keyword to (first sig). honeysql declares its InlineValue
protocol exactly that way — with the fix, all four honeysql namespaces
load unmodified from git and the formatter produces correct sqlvecs for
selects/inserts/updates/deletes/joins/:inline. Listed in libraries.md.
A native-executable build bakes the jolt ctx, and env-reading libraries
(config.core/load-env) snapshot the ENTIRE build environment into it — jpm
marshals that into the binary. GitHub push protection caught real API
tokens inside an example's build output this way.
With JOLT_BAKE_ENV_ALLOWLIST set (comma-separated names — a project's
build.sh exports it for the bake), System/getenv serves only the listed
variables: single-name reads of unlisted vars return nil and the full
snapshot is filtered. Unset — every normal runtime — reads stay live and
unfiltered, so a baked binary that re-reads env at startup (config.core/
reload-env) sees the real runtime environment.
Verified A/B on ring-app: a planted token appears in the unscrubbed binary
(strings | grep: 1 hit) and not in the scrubbed one (0), which still serves.
Direct janet.os/environ bridge calls remain unfiltered host access, as
documented.
Running a checkout's build/jolt-deps by path failed with ENOENT unless
build/ was also on PATH: exec-jolt spawned a bare "jolt". Resolution is
now $JOLT_BIN, then the jolt sitting beside this binary (argv[0]'s
directory — the pair is built together), then PATH.
The vendored spork/http is gone — jpm owns janet packages. In its place:
- The janet.* bridge autoloads jpm-installed modules on first reference:
janet.spork.http/server requires spork/http from the module path and
caches its bindings (failures are negatively cached). Works for any
module, in every mode, including inside net/server connection fibers.
- deps.edn grows a :jpm/module coordinate: jolt-deps verifies the module is
importable at resolve time, optionally running `jpm install` on the
:jpm/install package once when it isn't, and otherwise fails with the
install hint. Contributes no source roots. ring-app declares spork/http
this way.
Docs: README's interop section, docs/tools-deps.md (:jpm/module reference),
and the ring-app README (including the jpm-version caveat for spork HEAD's
.janet native sources, which older jpm rejects).
(^bytes [b])-style return hints reach fn as a (with-meta [b] {:tag ...})
form; unhint sheds the wrapper through the rebuild path so the clause
representation never changes. The host-interop hint rows exercise it.
Class names evaluate to their canonical class-name STRINGS (the same values
class returns), so a (defmulti m (comp class :body)) matches (defmethod m
String ...) — ring.util.request dispatches exactly this way. Constructor
sugar and the new special form resolve the actual ctor from the registry
when given a token; dispatch-only names (InputStream, File, ISeq, ...) are
interned for defmethod position. nil is a legal multimethod dispatch value
now (sentinel-keyed: janet tables drop nil keys) — ring keys body-string's
no-body case on it.
spork/http is VENDORED (vendor/spork/http.janet, MIT) and baked into the
image, reaching the jolt layer as janet.spork.http/* through the janet.*
bridge — whose lookup is now a chain (runtime fiber env, module env, the
vendored registry), which also fixes janet/* resolution inside net/server
connection fibers (they carry a foreign env). jolt.http is rewritten over
the spork client (its old net/request never existed; also fixes its own
get shadowing clojure.core/get).
Also: slurp accepts opts and DRAINS reader shims (ring middleware slurps
request bodies), clojure.string/replace takes fn replacements with Clojure's
match-or-groups argument, .indexOf int needles are char codes, .getBytes on
the String surface, and ^bytes-style return hints on param vectors parse
(the fn macro unwraps the with-meta form).
Suite steady at 4715/5348; conformance x3 green; deps-conformance medley +
cuerdas green (the stuartsierra/dependency failure predates this change).
The interop surface ring.util.codec needs (registered through the javatime
shim registries): URLEncoder/URLDecoder (www-form-urlencoded in pure janet),
Charset/forName, Base64 encoder/decoder, Integer/valueOf with radix +
parseInt, StringTokenizer, clojure.lang.MapEntry (a 2-tuple), a String ctor
from bytes, .getBytes on the String surface, and a java.lang.Number method
surface (byteValue and friends).
Protocol fixes: extend-protocol on java.util.Map/Set/List now dispatches
(maps — phm/struct/sorted/records — never produced host tags and fell to
Object), lazy seqs gained their ISeq tags, and a nil extension arm works
(group-by-head and extend-type both choked on the nil head). reduce
dispatches to a reified clojure.lang.IReduceInit's own reduce method, which
is how ring-codec tokenizes.
jolt-deps learns :deps/root (tools.deps monorepo subdirectory checkouts —
ring-core lives inside ring-clojure/ring). spork/http, when jpm-installed,
reaches the jolt layer as janet.spork.http/* through the janet.* bridge
(soft: nothing requires it unless used).
The protocol fixes alone let 29 more clojure-test-suite assertions execute:
5319 -> 5348 run, 4706 -> 4715 pass.
The first per-type migration print-method unlocked: uuid, regex, transient,
and channel rendering move from host pr-render branches to io-tier
defmethods (exact same output). The renderer's tagged fallthrough now
dispatches ANY remaining :jolt/* value through the print-method hook before
the raw pairs view — so every tagged type is user-overridable, atoms
included ((defmethod print-method :jolt/atom ...) fires nested), and future
per-type migrations are pure overlay additions.
Hot types (numbers, strings, symbols, collections) stay native, and inst/
namespace/var stay host for now — their formatters (rfc3339, display names)
live there anyway. A transient's :kind is read with jolt.host/ref-get: get
on a transient is the dispatched collection lookup (same trap as sorted
colls).
Before the hook is wired (init-time error messages) tagged values fall
through to the pairs view — bootstrap rendering only.
print-method/print-dup are now multimethods in the io tier with Clojure's
exact dispatch ((:type meta) keyword, else type — core.clj 3693). On jolt the
dispatch value for a record is its quoted full-name symbol, since class names
aren't values here.
Records used to pr-str as the raw janet table; the renderer's record branch
now prints Clojure's #ns.Type{:k v} syntax, and first consults a callback the
api wires up after the overlay loads — so a user defmethod on a record type
fires everywhere: top level, nested in collections, through pr/prn/pr-str.
Builtin overrides (a :number method) fire only on direct print-method calls;
pr keeps the native fast path (documented divergence).
java.io.Writer arrives as a shim beside the StringReader/StringBuilder ones:
a :jolt/writer tagged value with write/append/flush/toString, a StringWriter
ctor, and a sink variant the renderer callback uses.
Two latent host bugs fixed on the way: the interpreted syntax-quote splice
blew up on ~@nil (an interpreted macro's empty & rest binds nil — first tier
user of defmulti found it; d-realize now treats nil as the empty seq), and
(print-method x nil) now throws like the JVM instead of returning nil.
10 spec rows; bench dead even (sandwich run); greeter green on a fresh
binary.
Batch 2 of the post-shrink sweep, all pure compositions or documented stubs:
enumeration-seq/iterator-seq (seq), promise/deliver (an atom — deref of an
undelivered promise stays nil, single-threaded host), bean, uri?,
special-symbol? (an evaluated set of quoted symbols — a QUOTED set literal
stays an unevaluated reader form on jolt, which the first version tripped
over), print-method/print-dup (inert until jolt-g1r), and the whole proxy
surface (mappings/call-with-super/init/update pass through, the constructive
half throws). The seed loses 16 defns and bindings; nothing kept.
Three canonical-conformance fixes from the post-shrink batch:
- bit-and/bit-or/bit-xor/bit-and-not get Clojure's variadic arities as
20-coll shells folding the binary host ops (now __bit-* seams). 2-arg call
sites still compile to the native janet op via the backend's native-ops
table. The passes.clj constant-fold table now names the seams — the public
fns are overlay and don't exist when the compiler loads (this briefly broke
every compile-mode init).
- core-set? recognizes the :jolt/sorted-set representation (jolt-dpn):
(set? (sorted-set 1)) was false, and ifn? on sorted sets inherited the bug.
- (if) / (if test) / (if test then else extra) throw in both the analyzer
and the interpreter — spec 03-special-forms X1, now marked verified.
Suite 4704 -> 4706; bench and the greeter example benchmark are flat.
Fixed arities now throw Clojure's ArityException shape — 'Wrong number of
args (N) passed to: name' — on any count mismatch; variadic arities on fewer
than their fixed params. The compiled path already enforced fixed arities via
janet's native fn check and multi-arity dispatch; this adds the check to the
interpreter's single-arity closures (the oracle was silently dropping extra
args and giving a raw tuple-index error for missing ones) and guards the
compiled single-variadic wrapper's minimum. Messages carry the fn name when
there is one. 16 spec rows; the update.cljc suite row flipped green (4703 ->
4704).
Enforcement exposed that seq-to-map-for-destructuring had drifted: the spec
row called the 1-arity fn with two args, and the body silently dropped a
trailing unpaired element. Replaced with the canonical Clojure 1.11 version
(even pairs build the map, a single trailing element passes through — so
(f {:b 2}) kwargs calls work — and an unpaired key throws).
Also: transients RFC notes tuple support from the seed-shrink rounds.
Round 6 of the seed shrink (the printer round, scoped by the perf wall). The
five wrappers move to 20-coll over two new host seams: __write (push a string
to *out*) and __pr-str1 (render one value readably). The renderer itself
stays in the seed — it's representation-coupled (pvec/phm/phs/sorted
internals) and shared with the hot str, and rendering through overlay calls
would pay the per-element call cost everywhere big values get printed.
print-method as a real multimethod is follow-up work.
The new spec rows caught a renderer bug: string bodies were never escaped, so
(pr-str "a\"b") didn't round-trip through the reader. pr-render now
escapes quote/backslash/control chars per Clojure.
Round 5 of the seed shrink. transduce is the canonical 5-liner over reduce
(which already honors reduced and steps lazy seqs); eduction composes with
comp and stays eager into a vector (documented divergence, as before);
td-comp — eduction's last caller — is deleted from the seed. transient
accepts tuples now (reader vectors / map entries), so (into [] (first {:a 1}))
keeps working everywhere a vector does.
into was moved, benched, and moved back: the overlay call layers cost the
into-vec suite ~11% back-to-back (536 vs 480ms), the same per-call wall that
sent even?/odd? home in round 4. A transient conj! fast path didn't pay for
itself either (jolt call overhead dominates, not the per-element conj). The
seed keeps core-into + its private transduce machinery; the binding count
still drops by three.
Round 4 of the seed shrink. zero?, pos?, every? move to the syntax tier
(empty? and the analyzer use them — raw def+fn* per the file constraint);
char? joins the tagged-value predicates in 20-coll. coll? stays seed: host
set? doesn't cover sorted sets (filed jolt-dpn) and the tag check from the
overlay would hit the sorted-coll get trap. pos? guards number? explicitly —
the staged recompile emits bare > as the native janet op, which orders
strings (zero? gets the same guard; spec rows lock both plus neg?).
The canonical every? seq-walks its coll, which exposed that rest/next over
sets, phms, struct maps and sorted colls fell into core-rest's indexed
fall-through and walked the wrapper table's INTERNAL fields — (next #{1 2})
was (nil nil), (clojure.set/subset?) broke. core-rest now seqs those
representations (branches placed AFTER the hot vector/lazy paths; the first
ordering cost seq-pipe 4x). Suite rises 4700 -> 4703; baseline 4660 -> 4695.
even?/odd? are back in the seed after the bench A/B: (filter even? ...) pays
an extra call layer per element through the overlay (seq-pipe 262 -> 1100ms).
They join the perf-wall list with the lazy hot fns.
Round 3 of the seed shrink. To the overlay: identity, constantly, neg?,
even?, odd? (20-coll, ahead of their first in-tier uses), not= and unreduced
(00-syntax — the kernel and seq tiers use them), ==, ensure-reduced,
halt-when, parse-boolean, parse-uuid, newline, seque, array-seq, to-array-2d,
and the masking unchecked-byte/short/char/float/double coercions. parse-uuid
validates via re-matches over a new __make-uuid host binding (overlay source
can't write :jolt/type map literals). memfn moves to 30-macros as a working
macro over the .method call sugar instead of a fn that throws.
Behavior fixes toward Clojure, each with spec rows: == now throws on
non-numbers instead of comparing them, and halt-when is the canonical
::halt-map version (the halt value replaces the whole reduction result, no
double completion). list? and map-entry? stay in the seed — both are
representation-coupled (plist/tuple checks).
clojure-test-suite goes 4701 -> 4700: update.cljc expects
(update {:k 1} :k identity 1 2 3 4) to throw an arity error, and jolt fns
don't enforce fixed arity anywhere (pre-existing, language-wide — the seed's
Janet identity threw natively). Filed as jolt-6xn; fixing it should flip
several suite rows at once.
Jolt numbers don't overflow, so +'/-'/*'/inc'/dec' and the whole unchecked-*
family are just the checked ops — now one-line defs in core/20-coll.clj
instead of ~25 seed bindings. int? and num move the same way.
unchecked-divide-int now goes through quot, so dividing by zero throws like
the JVM instead of silently truncating infinity. unchecked-int/long gain
char handling via int, matching Clojure ((unchecked-int \a) => 97). The
masking byte/short/char coercions are not aliases and stay in the seed for
a later round.
Also drops a second duplicate set of unchecked defns that was shadowing the
first at module load.
The seed copies of inst?/inst-ms and the multimethod table ops
(get-method/methods/remove-method/remove-all-methods/prefer-method) are dead
code — the overlay redefines all of them (20-coll.clj, 30-macros.clj) — so
they're deleted along with duplicate bindings-table entries (unreduced,
eduction, unchecked-inc/dec/add/subtract).
New spec rows for methods/remove-all-methods caught two real bugs in the
evaluator's setup fns: methods-setup returned the live host table (count
rejected it, and callers could mutate dispatch state), and
remove-all-methods-setup swapped in a fresh table the dispatch closure never
saw. methods now returns a phm snapshot; remove-all-methods clears in place.
Loading these libs via require worked (load-ns-source interprets, macros
expand lazily) but the same code inlined by uberscript routes through
eval-toplevel and compiled, surfacing four gaps:
- a ^{:map} metadata def name reads as (def (with-meta name m) v); the
analyzer died extracting the name (config.core's defonce env). It now
throws uncompilable so the interpreter, which handles it, takes over.
- declare was a no-op, so a compiled forward reference to a declared
name that collides with a janet root binding bound to the host fn
(selmer.parser's (declare parse) compiled to janet's 1-arg parse).
declare now expands to no-init defs, the interpreter interns them,
and the analyzer routes no-init def to the interpreter.
- class? was missing (selmer.util's exception macro calls it at
expansion time). Always false, like ratio? — no Class objects here.
- require of an unlocatable namespace silently left an empty ns behind,
deferring the failure to an unresolved symbol far from the cause. It
now throws like Clojure's FileNotFoundException. Namespaces entered
in-session count as loaded (Clojure puts them in *loaded-libs*), and
the SCI bootstrap opts out via :lenient-require? since its
clj-targeted requires can't all exist on this host.
The cache key was built with janet's (hash ...), which is seeded per
process, so it never matched across invocations and every jolt-deps run
re-resolved and re-fetched. Store the raw key material instead.
Run the jpm git calls silenced (:silent) with a one-line progress note
on stderr — the checkout chatter ("HEAD is now at ...") was landing on
stdout and corrupting the documented JOLT_PATH=$(jolt-deps path) capture.
Covered by two new deps-resolve checks: a cross-process cache-hit probe
(sentinel-tampered cache file) and a stdout-cleanliness check against a
local file:// git dep.
yogthos/config now loads and runs end to end: config.edn/.lein-env
deep merge, env vars keywordized and type-converted, PushbackReader
over io/reader, reload-env via alter-var-root. New shims:
java.io.PushbackReader (read/unread), Long/parseLong, BigInteger.,
Boolean/parseBoolean, System/getProperties; clojure.edn/read now
drains an actual reader (it used to read one LINE from a raw janet
file handle, so multi-line config files and shim readers both broke).
Three real bugs shaken out, each with regression specs:
- An empty rest arg bound () instead of nil. ((fn [& r] (if r 1 2)))
returned 1; the truthy () sent config's (or (.exists f) required)
down the wrong branch. Fixed in the interpreter and the compiled-fn
emission. Internal apply boundaries (protocol dispatch, core-apply)
now accept a nil seq like Clojure's (apply f x nil).
- seq/map over a raw janet table (System/getenv, os/environ) yielded
nothing, so config's read-system-env came back empty. Raw host
tables now seq as kv entries like any map, in core-seq,
realize-for-iteration, and coll->cells. The old spec row hid this
behind a vacuous (every? pred empty) — replaced with one that
asserts non-emptiness.
- edn/read single-line limitation, as above.
test/integration/config-lib-test.janet runs the real library from
~/src/config (skips when absent).
Git clones now default to a shared, sha-immutable cache —
$JOLT_GITLIBS, else <config-dir>/gitlibs — instead of a per-project
./jpm_tree, the tools.gitlibs ~/.gitlibs model. Passing tree
explicitly still works (tests do). The resolved-roots cache moves
out of the clone tree to the project-local .cpcache/jolt-deps.jdn,
since roots depend on the project while clones don't.
deps.edn grows :tasks, the honest subset of babashka's: a string
task is a shell command, a map task is {:main-opts [...] :doc}.
jolt-deps tasks lists them (merged user+project), jolt-deps task
NAME runs one. Bare-expression tasks are out of scope: the reader
hands back parsed data and round-tripping to source is fragile.
Also fixes load-config skipping the symbol-key normalization when
only one config file existed — :tasks/:deps keys stayed raw reader
symbols (which embed positions and never compare equal), so lookups
missed. Regression rows in deps-tasks-test; docs updated for the
whole tools.deps surface (aliases, -A/-M, user config, conflicts,
gitlibs cache, tasks).
deps.edn :aliases now work the tools.deps way, scoped to what jolt
supports (git/:local, no maven): :extra-paths and :extra-deps
accumulate across selected aliases, :main-opts is last-wins. The CLI
grows -A:dev:test (selects aliases for path/run/repl/-e) and
-M:alias (runs the alias :main-opts through jolt). A user-level
deps.edn ($JOLT_CONFIG, else $XDG_CONFIG_HOME/jolt, else ~/.jolt)
merges under the project file: :deps and :aliases merge per key with
the project winning, everything else replaces.
Resolution is now breadth-first so a top-level coordinate always
beats a transitive one for the same lib (it was DFS first-wins —
a dep's pin could shadow the project's own). Conflicting coordinates
for one lib warn on stderr with both coords and which won. Also
fixes dedup keying: it hashed the symbol struct's string repr, which
carries reader position metadata, so the same lib from two files
never deduped.
resolve-deps-cached keys on project edn + user edn + aliases.
Tests: deps-aliases-test and deps-conflicts-test, local deps only.
Selmer now loads and renders templates on jolt: variables, filters
(upper, date with JVM patterns), if/for tags, nested lookups, HTML
escaping, and render-file with its last-modified template cache.
New src/jolt/javatime.janet provides the java.time surface Selmer's
date filters use (DateTimeFormatter/Instant/ZoneId/LocalDateTime/
FormatStyle/Locale, epoch-ms backed, host-local timezone) plus the
java.io/java.lang/java.net shims its template reader needs
(StringReader, StringBuilder, URL, File/separator, Class/forName).
Everything registers through three new evaluator registries
(class-statics, tagged-methods, class-ctors), so the module is data
plus an install call.
Fixes shaken out along the way, each load-bearing for Selmer and
correct on their own:
- :refer :all silently referred nothing (it iterated the :all keyword)
- ns :import ignored vector specs and didn't share deftype ctor vars
- dot calls on deftype/reify instances never consulted the protocol
registry, so (.render-node node ctx) failed where (render-node ...)
worked
- instance? rejected expression type args like (Class/forName "[C")
- char-array didn't accept a string
- io/resource now searches the loader's source roots (the classpath
analog); io/reader handles char arrays, URLs, readers, and returns
an in-memory reader with :read-line-fn for file paths
- String .split (regex, JVM trailing-empty semantics), file-path
methods (.toURI/.toURL/.getPath/.lastModified/.exists)
- System/getProperty (os.name & co), the janet/* bridge now works
inside env-less fibers, and qualified class names that syntax-quote
mangles (selmer.util/StringBuilder) fall back to the ctor registry
Spec rows cover the shim surface; test/integration/selmer-test.janet
runs the real Selmer from ~/src/selmer (skips cleanly when absent).
\p{L}/\p{Lu}/\p{Ll}/\p{N}/\p{Z}/\p{Ps}/\p{Pe} (+\P negation) land in
both escape positions of the regex compiler, mapped onto the byte PEGs:
ASCII exact, any high byte (inside a UTF-8 sequence) counts as a LETTER —
so ^\p{L}+$ accepts UTF-8 words while \p{N}/\p{Z} stay ASCII. (?u) was
already a tolerated no-op flag. Unknown property names error at compile.
Chasing the acceptance target (cuerdas via deps-conformance) pulled in the
rest of its clj-compat chain, each a real gap:
- the deps-conformance harness reads libraries under clj-compat reader
features (deps are clj/cljc by definition — without :clj, cuerdas's
#?(:clj (instance? Pattern x)) branches resolved to NIL bodies)
- instance? knows Pattern/java.util.regex.Pattern (regex values) and
Character (cuerdas's rx/regexp? gate on split)
- the java.lang.String method surface: .toLowerCase/.toUpperCase/.trim/
.indexOf(-1 on miss)/.lastIndexOf/.substring/.charAt/.startsWith/
.endsWith/.contains/.replace/.equalsIgnoreCase/... — ASCII case mapping,
unknown methods error (the old path silently returned nil)
- the (.method obj args) SUGAR now desugars to (. obj method args) in the
interpreter — it was never implemented (bare .method heads resolved as
vars, hence 'Cannot call nil')
- Long/MAX_VALUE / MIN_VALUE statics (f64 approximations)
deps-conformance: medley ok, cuerdas ok (was check-error); dependency now
loads its clj branches and fails only on its single-segment ns resolution.
30 new spec rows (11 regex, 19 interop). Gate exit 0.
ifn? (jolt-1vx) is the canonical IFn set in the overlay: fns, keywords,
symbols, maps (sorted included), sets, vectors, and vars — NOT lists. The
seed version said true for lists and false for struct maps and vars.
Mutable-mode caveat documented (vectors and lists share the array repr
there). 13 predicate rows.
Multimethod dispatch (jolt-heo) now collects EVERY isa-matching method key
and picks the dominant one — x dominates y when prefer-method'd over it or
(isa? x y) — and two matches with no dominant is an ambiguity ERROR, as in
Clojure. It used to take whichever key the table yielded first, silently
ignoring prefer-method. The prefers store upgrades to Clojure's
{x -> set-of-dominated} shape, shared between the dispatch closure and
prefer-method-setup via the var; prefers becomes a macro over a setup fn
(the store lives on the VAR — the multifn value can't carry it, so the old
fn read {} forever). 6 multimethod rows + the conformance row updated to
the canonical shape (335x3).
The reader (jolt-ou8) kept the pending KEY when a comment or #_ sits in a
map's VALUE slot: the old code dropped both, desyncing kv pairing — the
real value became the next key and the closing brace landed in value
position ('Unmatched closing brace'). Selmer's deps.edn (a '; for
development (REPL, etc)' comment between key and value) now parses; 6
reader rows incl. nested commented maps.
Gate: jpm exit 0, conformance 335x3, all tests passed.
jolt.passes is the new portable pipeline stage between the analyzer and the
back end: pure IR -> IR rewrites, total over node :ops (unknown ops pass
through with folded children), loaded with the compiler namespaces and
resolved lazily by analyze-form (JOLT_NO_IR_PASSES=1 disables — the same
escape-hatch pattern as the macro oracle). The shape is flatiron's opt.clj
applied to the jolt IR, which is what jolt-2om asked for.
The first pass is constant folding: a call of a foldable numeric SEED fn
(the later tiers don't exist when the compiler loads) whose args are all
constant numbers becomes a constant, and an if with a constant test becomes
the taken branch (dead-branch elimination — the untaken side never even
resolves). Folding computes with the ACTUAL jolt fns, so results match
runtime semantics by construction; a fold that would throw (mod 5 0) is
left for runtime.
Two walk lessons paid for in debugging: let/loop bindings are
[name init-ir] PAIRS, not maps (assoc'ing :init into a pair corrupts it);
and a throw inside the interpreted pass unwinds past the interpreter's ns
restores, so analyze-form restores the compile ns after the (protected)
pass call — without that, one pass error left current-ns in jolt.passes and
the rest of the tier compile resolved against the wrong namespace (sort-by
landed on the 2-arg JANET builtin).
ir-passes-test pins folds, conservatism (free vars, throwing folds), and
end-to-end eval. Gate exit 0.
Every walk over a lazy seq created FRESH wrapper tables around the shared
rest-thunks (ls-rest, ls-seq/ls-count, realize-for-iteration, the printers,
reduce — each had its own make-lazy-seq loop), so independent walks re-ran
the thunks: side effects duplicated, and a doall'd seq of futures was
re-spawned serially by the deref walk. Every walker now goes through
ls-rest-cached, which memoizes the rest wrapper on its node — thunks run
exactly once, as in Clojure. Costs ~10% on walk-heavy benches (the per-node
cache get/put — Clojure's LazySeq pays the same); net still -9% vs the
pre-linear-walks baseline. Three regression rows pin once-only effects and
value stability across walks.
On top of that: pmap/pcalls/pvalues (jolt-oeu) over the real-thread futures
— spawn-all-then-deref (the once-only fix is what makes the doall actually
mean that), snapshot semantics documented, multi-coll arity via the
canonical vector-zip. System/currentTimeMillis + nanoTime land as System
statics (the realtime clock — os/time is whole seconds, which quantized
every elapsed measurement to 1000ms). Seven pmap rows incl. a generous-
margin parallelism check (4 x 200ms sleeps under 700ms after warmup).
The native-ops table grows from 9 to 16, each verified for semantic parity
with the jolt fn before inclusion (incl. negative operands): mod is floored
on both sides, rem (janet %) truncates, / is variadic with (/ x) -> 1/x.
quot is deliberately absent — janet div floors where Clojure truncates.
jolt's bit fns are 2-arg (unlike Clojure's variadic), so the bit ops emit
native only at exactly that arity; bit-not is unary. Eight new conformance
rows pin compile=interpret on the new ops at their guarded arities (334x3).
map-read 10.8 -> 9.2 ms (the (mod i 100) in its loop inlines). From the
flatiron review's unchecked-primitive-loops idea.
Reviewing flatiron's morsel batching for applicability turned up something
better hiding under the lazy machinery: every cell over a concrete
collection was produced by slicing the REMAINDER ((tuple/slice c 1) per
element in coll->cells, and rest/next sliced the same way), so any full walk
was O(n^2). mapv over 40k elements took 10.4 SECONDS; a 20k-element
first/rest loop took two.
Cells over indexed collections now walk by INDEX (one shared indexed-cells
helper, O(1) per step), and rest/next of a vector/tuple/list return an O(1)
lazy view from index 1 — which also makes (rest [1 2 3]) a SEQ, as in
Clojure (it was a vector-typed slice; seq?/vector? rows pin the change).
mapv 40k: 10405 -> 182 ms. rest-loop 20k: 2040 -> 31 ms. Whole bench:
seq-pipe -28%, into-vec -24%, str-join -18%, hof -26%, TOTAL 4565 -> 3753
(-18% vs main, back to back). Chunked seqs (jolt-yqc) drop in priority:
the quadratic walks were the actual cost; chunking now only amortizes
per-element closure allocation.
Nine regression rows incl. 20k/50k linear-scaling smoke tests. Gate exit 0.
require-:as wrote the string-keyed :imports table (which resolution reads)
while ns-aliases read the symbol-keyed :aliases table (which nothing wrote)
— so (ns-aliases) was always empty and the alias fn had to write both as a
bridge. :aliases (alias-name string -> ns-name string) is THE store now:
require :as and the alias fn write it, both resolution paths read it first
(falling back to :imports for class imports, which is all that table holds
now), ns-unalias removes one entry, and ns-aliases presents Clojure's
{alias-symbol -> namespace object} shape built from it. ns-resolve's
qualified path goes through the same lookup.
Also: the coverage dashboard's last 'resolvable-not-interned' entry was '.'
— which (resolve '.) returns nil for on the JVM too; the tool now classifies
it as the special form it is, and that category reads ZERO.
7 new unified-alias spec rows (require/alias/ns-unalias round-trips through
both the resolution and introspection views); the white-box namespace test
tracks the accessor rename. Gate exit 0.
(recur (inc acc) (rest xs)) re-entered the fn through its varargs collector,
so the rest seq came back wrapped in a fresh 1-element rest list — xs never
emptied and the interpreter hung (jolt-4df; the compile path was already
correct). recur now re-enters through a dedicated entry that binds the LAST
arg directly as the rest param (n-fixed + 1 args, Clojure's contract), in
both the single-arity and multi-arity fn* paths; the shared body runner keeps
the ns-swap/restore in one place, and fixed arities still re-dispatch through
the arity dispatcher exactly as before.
Six spec rows: the original repro, zero-fixed variadic, rest-empties-to-nil,
multi-arity variadic recur, nil rest, and a fixed-arity control.
clojure.edn was nearly complete (sets, #uuid/#inst, :eof all landed earlier);
the :readers opt was ignored and :default missing. Both work now — the
reader stores a tag as a :#name keyword, so the lookup normalizes it to the
symbol Clojure keys :readers with; :default gets (tag value); built-in data
readers stay the fallback. 8 new edn spec rows. This was the last open item
of jolt-0mb (the vendored walk/zip/data/edn battery has been green for a
while: 34/33/61/50, all clean).
Chasing the probe cascade ('Unable to resolve symbol: edn/...' after one
error) found a real evaluator bug: an interpreted fn body runs with
current-ns rebound to its DEFINING ns and restores it with a plain trailing
call — an UNCAUGHT throw skips every restore on the way out, leaving the ctx
stuck in the deepest callee's ns, where alias-qualified lookups then fail
(the same cascade previously seen via sci). The repair lives at the
TOP-LEVEL boundary (loader/eval-toplevel saves the entry ns and restores it
on error before re-raising) — NOT per-call defer/try, which builds a fiber
per frame and blew the C stack on deep interpreted recursion (file-seq)
when tried first. Regression tests cover the cross-eval leak and that
aliases keep resolving.
test/spec/untested-vars-spec.janet adds 143 rows asserting jolt's documented
behavior for the whole implemented-untested category — primed arithmetic,
the array/aset/coercion stubs, unchecked-*, the chunk family, JVM-shape
stubs (class/bean/proxy/memfn as resolve-only or :throws), ns/REPL
machinery, and the misc seqs. tools/spec_coverage.py now checks each var as
a whole TOKEN in the test sources (call-position-only matching missed *1,
+', ., .., /, and bare transducer refs like cat).
Writing rows from probed truth surfaced five real bugs, all fixed:
- comp with a jolt-IFn stage silently returned nil ((comp seq :content)) —
raw Janet keyword application is not jolt invoke. comp is the canonical
overlay defn now (fixed-arity composed fn, so the hot 1-arg path is two
direct calls); the seed keeps a private td-comp only for the transducer
machinery. hof bench +9% vs native, the price of correct IFn dispatch.
- extend (the fn) was a nil-expanding stub MACRO shadowing any definition;
it's a real fn over register-method now, and extends? (a constant-false
stub) is real over extenders
- (.. x f g) hit the 'ClassName.' constructor branch (a name ending in a
dot) and died; .. is the canonical threading macro now
- aclone errored on pvecs; ns-interns/ns-imports returned live host tables
that count/seq reject (now structs)
Thread/sleep + Thread/yield land as Thread statics beside Math/: sleep parks
the WORKER's own event loop (each future thread has one), which makes timed
deref provably fire — futures-spec gains the timeout-fires, sleep-in-body,
and timed-out-future-still-completes rows. The futures impl itself already
ran on real OS threads (ev/spawn-thread + marshalled results); jolt-ejx was
stale.
Dashboard: implemented+tested 433 -> 564 of 694; implemented-untested and
missing-portable are both EMPTY. Gate: jpm exit 0, all tests passed.
The dashboard's missing-portable category is now EMPTY (was 35 when the issue
was filed; this session's io/leaf work had already landed most of them).
The final seven:
- extenders — ctx-capturing clojure.core fn over the protocol type-registry:
the type-tags implementing a protocol, as symbols; nil when none
- find-keyword — keyword: jolt keywords have no intern table, so it always
finds (babashka makes the same call)
- inst-ms* — the raw Inst method; one inst representation, so = inst-ms
- read+string — over the 50-io readers, which now expose :buf and :fill-fn;
returns [form exact-text-consumed], EOF throws or yields [eof-value ""]
with the 3-arity, works for string AND stdin readers
- with-local-vars — fresh free-standing var cells (__local-var seam) bound as
locals; var-get/var-set work on any cell
- with-open — canonical recursive expansion closing through the __close seam:
a map-like value's :close fn or a host file (no .close interop here);
nested closes run inner-first, finally runs on throw
- with-precision — body evaluates with precision/:rounding accepted and
ignored (doubles, no BigDecimal context) — documented divergence
30 new spec rows (test/spec/missing-vars-spec.janet); coverage.md
regenerated: implemented+tested 426 -> 433, missing-portable 7 -> 0.
Gate: jpm exit 0, all tests passed.
The phm had a FIXED 8 buckets, so a 100-entry map was a ~12-entry linear
scan per lookup — and phm-get walked the bucket twice (contains? then find).
This went mostly unnoticed until the canonical zipmap (batch 2) started
returning phms where kvs->map had built structs for scalar keys, regressing
the map-read bench ~7x (jolt-s3y).
phm-assoc now rehashes into a doubled bucket array when the count passes 2
entries/bucket (done on the fresh copy, so persistence is untouched);
phm-get is single-pass with a presence flag (nil values still distinguish
from missing); key= tries identity/scalar equality before paying for
canonicalization; the bucket count is derived from (length (m :buckets)),
not a constant, so any already-marshaled map keeps working. core-contains?'s
phm branch goes through phm-contains? instead of poking buckets directly.
map-read 48.5 -> 10.9 ms (the residual vs the pre-batch-2 6.7 is the
canonicalizing-representation constant); map-build steady; bench TOTAL 4457
vs 4565 on main back-to-back. New unit case crosses the resize boundary at
500 entries: every key found, nil values present, collection keys canonical,
dissoc + persistence intact. Gate: jpm exit 0, conformance 326x3.
nrepl-test was CI's only failure: the server subprocess ran main.janet from
source, paying the full compile-mode init, which outran the 5s connect poll
on slow runners (locally it always won the race). The test now prefers
build/jolt — its ctx is baked at build time, so it accepts in ~20ms and CI
builds it anyway — falls back to source, polls up to 60s under a 90s
watchdog, and dumps the server's stderr when startup fails so the next CI
failure is diagnosable.
CLAUDE.md's placeholder sections become real: build/test commands with the
run-the-gate-with-a-real-exit-code protocol (a piped gate once shipped masked
spec failures), the seed/overlay/tier architecture sketch, and the porting
gotchas that have each bitten at least once (leaf verification, stub-breaks-
self-recursion, tier macro ordering, ref-get vs get on attached-ops wrappers,
:jolt/type map keys, expander-called fns, canonical-port policy) — previously
only in local bd memories.
An indirect global reference emitted ((var-get 'cell) ...) — a function call
per deref, whose body is a binding-stack check plus a root read. The emitter
now inlines that: (if (in 'cell :dynamic) (var-get 'cell) (in 'cell :root)).
Non-dynamic vars — the vast majority of references — pay two native table
ops and a branch instead of a call; dynamic vars take the full var-get
(thread-binding walk). Redefinition stays live (:root is read per call) and
binding semantics are exact: the :dynamic check is PER CALL, not at emit,
because a (def ^:dynamic x) in the same compiled unit marks the cell dynamic
only when the def runs — the same reason JVM Clojure's Var.deref() checks
the thread-bound bit every call (an emit-time variant was 1.7x faster still
on fib but failed conformance exactly there).
fib 130 -> 74 ms (1.75x); bench TOTAL 4564 -> 4437 back-to-back. This
displaces the gen-counter inline-cache design from jolt-8sq: with var-get's
existing fast path, resolution was never the cost — the call was. A
gen-guarded cache would add state per site to save nothing further, and
couldn't skip the dynamic check anyway.
Found while benching: map-read regressed ~7x back in batch 2 (canonical
zipmap builds a phm where kvs->map built a struct) — filed as jolt-s3y.
Gate: jpm test exit 0, conformance 326x3, suite >= baseline.
sort-by, rand-int, shuffle, random-uuid, char-escape-string, and
char-name-string move to 20-coll over the two host seams that stay (rand and
sort — they ARE the randomness/ordering primitives). Canonical upgrades ride
along: sort-by defaults its comparator to compare, so nil sorts FIRST (the
kernel fn used host ordering and put nil last); rand-int truncates toward
zero via int (the kernel fn floored, wrong for negative n); shuffle is a
pure-functional Fisher-Yates over vector assoc and rejects non-collections
(a string is seqable but not shuffleable, as on the JVM — the honest gate
caught that one); random-uuid builds over rand-int and validates through
parse-uuid; the char tables are char-keyed Clojure maps (Clojure's shape —
the seed keeps its private code-keyed copies for pr-render).
22 new spec rows. Gate: jpm test exit 0 verified, suite 4698 >= 4660, bench
parity with main back-to-back (4733 vs 4817).
recompile-defns! is the defn analog of recompile-macros!: pre/at-kernel
overlay defns (00-syntax's destructure and friends; the kernel tier too in
interpret mode) load as interpreted closures, the evaluator stashes their fn
source on the var (:defn-src, scoped by a flag only api/load-core-overlay!
sets), and the end-of-init pass compiles them and swaps the var root. With
that in place, keys/vals/empty? — the fns the 00-syntax expanders call at
expansion time — move to the top of 00-syntax as raw fn* defs (canonical:
keys/vals project (seq m), so sorted maps come back in comparator order and
(keys {}) is nil; empty? keeps O(1) count dispatch with seq's cell check only
for the lazy/list fallback). The sorted tier drops its now-dead :keys/:vals
ops.
Correctness fixes that surfaced once the gate was run with a REAL exit code
(the previous 'jpm test | grep' gates reported grep's exit and masked spec
failures across #48-#50):
- map conj is strict again: a non-nil/non-map arg must be a 2-element vector
('Vector arg to map conj must be a pair'), and merge inherits it — the
batch-2 canonical merge had silently dropped the validation
- conj onto a lazy seq prepends (it fell into the MAP fallback); upstream
clojure.data/diff relies on (conj seq x) via set/union over keys, so diff
now matches Clojure exactly
- (seq {}) / (seq #{}) / empty phm are nil, not ()
- key/val are strict (a plain vector is not an entry); find mints a REAL
entry as the first entry of a one-entry map, nil values intact
- the sci avoid-method-too-large stub passes its registry map through
instead of returning a raw host table (strict conj rejected it; sci's
clojure-core registry is also no longer discarded)
Test updates: lazy-infinite pins take-nth realization at 5 (was 7 — the
canonical lazy impl realizes fewer); self-host asserts the analyzer IS loaded
in interpret mode (compiled expanders, PR #50) and is NOT in the
:compile-macros? false oracle. 18 new maps-spec rows.
Gate: jpm test exit 0 (verified directly, not through a pipe), conformance
326x3, suite 4698 >= 4660.
Macros are ordinary compiled fns in Clojure's model; compile mode has had
that since the staged bootstrap, but interpret mode — the conformance
battery's default — kept interpreted expanders, so every distinct (and ...)/
(cond ...) call form, and every fresh form produced by a recursive expansion,
ran an interpreted closure. ensure-macros-compiled! now runs in every mode:
interpret-mode init loads the tiers fast-interpreted, then one pass at the
end builds the analyzer (which itself stays interpreted there) and compiles
all stashed expanders; user defmacros after init compile too. The new
:compile-macros? opt (JOLT_INTERPRET_MACROS=1) preserves the fully-
interpreted oracle, and joins the ctx-image cache key.
Battery: 4700 pass / 90 clean files / 7 timeouts, from 4577 / 87 / 9 — two
macro-heavy files stopped timing out and 149 more assertions execute. The
compiled-expander delta proper is +67 passes (oracle mode on the same tree
measures 4633). Baselines raised 4540->4660, clean 86->88. Interpret init
grows 0.12s -> 1.12s for the analyzer build; init-cached amortizes it to ~5ms
per process.
New macro-expansion-test pins: expanders compiled in interpret mode (core +
post-init user defmacros), uncompilable bodies fall back interpreted and
still work, compile mode unchanged, oracle opt-out honored.
Follow-up filed (jolt-4j3): the same staged-recompile treatment for early
overlay DEFNS, which is what still pins keys/vals/empty? to the seed.
empty, assoc-in, and update-in move to 20-coll.clj as the canonical recursive
ports; interpose and take-nth move to the lazy tier WITH their canonical
transducer arities (volatile-based), so the seed's td-interpose/td-take-nth
helpers go too. (empty lazy-seq) is () now — the kernel fn returned a bare
host table for it.
keys/vals/empty? stay put for now: they're expander-coupled — 00-syntax's
when/and/or/cond/destructure expanders call them at expansion time, which
happens during the kernel-tier compile, before any later tier exists. They
move when early defns get the staged-recompile treatment macros already have.
26 new spec rows (incl. transducer arities through sequence/into and laziness
checks against (range)). Gate green: conformance 326x3, suite >= baseline,
full jpm test.
key/val/select-keys/zipmap/merge/merge-with/get-in/memoize/partial/
trampoline/some?/true?/false?/max/min/reverse move to 20-coll.clj as the
canonical Clojure definitions, plus find — which was previously missing from
jolt entirely (select-keys/merge-with/memoize build on it). Two behavior
fixes ride along: memoize now caches nil results (the kernel fn re-computed
them — canonical find-based impl), and conj of nil onto a map is a no-op as
in Clojure (it errored; the canonical merge relies on it). max/min keep the
JVM NaN behavior by construction (pairwise >/<). not= stays: the kernel tier
(subvec) uses it.
One new tier-ordering rule, learned the hard way: a tier may only use macros
from tiers that load BEFORE it — memoize's if-let (30-macros) broke compiled
init while interpret mode passed, because compile expands macros at tier
load and the interpreter expands lazily. Now documented in the migration
workflow note.
MIGRATION.md is gone — task tracking lives in beads (jolt-ded; the per-batch
workflow, tier-order rules, perf wall, and remaining candidates are in bd
memory core-migration-workflow). The doc's candidate lists had gone stale
against the actual seed anyway.
43 new spec rows. Gate green: conformance 326x3, suite >= baseline, full
jpm test, bench at parity with main back-to-back (4851 vs 4831 TOTAL).
Nine more seed leaves move to 20-coll.clj (verified leaf-by-leaf: defn +
core-bindings entry only, no internal callers). fnil is upgraded to Clojure's
canonical 2/3/4-arity — it patches only the first 1-3 arguments; the old
kernel fn patched every position it had a default for, which Clojure does
not. The rest carry their kernel semantics over unchanged (bigdec is a
double, numerator/denominator throw, supers is #{}, munge rewrites dashes).
16 new spec rows incl. the fnil arity-contract cases. Gate green:
conformance 326x3, suite 4577, full jpm test (2:18 — first full run with the
ctx image cache on main).
*in*, read-line, read, with-in-str, and line-seq land as a new overlay IO
tier (core/50-io.clj). *in* is a dynamic var holding a reader — a plain map
of two closures, :read-line-fn (next line, nil at EOF) and :read-fn (next
form, advancing past exactly that form). The default *in* reads real stdin
with a shared leftover buffer, so read and read-line interleave correctly;
with-in-str rebinds *in* to a string reader over one atom-held buffer —
(read) consumes its form, a following (read-line) returns the rest of that
line, as in Clojure. read has the 0/1/3 arities (EOF throws, or returns
eof-value when eof-error? is false).
The Janet seed grows two seams next to read-string: __stdin-read-line (one
line off stdin, newline stripped) and __parse-next (one form off a string ->
[form rest], nil at end of input) — and loses the line-seq stub.
Two traps hit and documented for future tiers: a map LITERAL with :jolt/type
as a key is read as a tagged form (don't tag overlay value maps), and a
leftover seed stub holding the same name breaks direct-linked self-recursion
— the overlay line-seq's recursive call bound to the stub's root, truncating
after one line. The stub's string-splitting behavior is kept as a documented
extension.
20 new io-spec rows (read-line EOF/interleave, read arities + eval round-trip,
line-seq incl. real-stdin paths). Gate green: conformance 326x3, suite 4577,
full jpm test.
sorted-map/sorted-map-by/sorted-set/sorted-set-by/sorted?/sorted-map?/
sorted-set?/subseq/rsubseq now live in their own overlay tier (25-sorted.clj).
A sorted coll is a tagged host table with a comparator-ordered :entries
vector, a 3-way :cmp, and the tier's op implementations ATTACHED to the value
(:ops map): the seed's conj/assoc/get/seq/count/... branches are each a
one-line call through (coll :ops), so the ops travel with the value — correct
across contexts, forks, and AOT images, no module-level hooks to re-wire.
The host surface grows by three minimal value primitives: jolt.host/
tagged-table, ref-put! (already there), and ref-get — a raw field read,
because plain get on a sorted coll IS the comparator lookup and reading
:entries with it recurses.
This fixes a pile of Clojure-correctness gaps the Janet kernel had:
- lookup/membership now go through the COMPARATOR: (contains? (sorted-set 1)
1.0) was a deep= scan, (conj (sorted-set 1) 1.0) and assoc of a
comparator-equal key now no-op/replace as in Clojure
- equality is representation-agnostic: (= (sorted-map :a 1) {:a 1}) and
(= (sorted-set 1 2) #{1 2}) were false
- iteration was broken: (map inc (sorted-set 3 1 2)) errored
(realize-for-iteration and coll->cells had no sorted branches)
- empty?/empty saw the host wrapper, not the collection: (empty? (sorted-map))
was false, (empty sc) returned a bare table; it now keeps the comparator
- sorted colls canonicalize as map keys; comparator fns may be boolean
predicates or 3-way (Clojure's fn->comparator)
- sorted-map throws on odd kv count; conj nil is a no-op
Also fixes jolt-h86 en passant: into-conj had no branch for sets (or sorted
colls) and silently returned the target unchanged — (into #{} [:a :b]) was
#{}. The fallback now folds conj. Regression rows in sets-spec.
sorted-spec grows to 77 rows (comparator-based membership, equality,
empty/rseq/printing, seq-fn interop, subseq/rsubseq on maps). Gate green:
conformance 326x3, suite 4577 (vs 4566 prior — the battery gained rows),
sorted+sets specs, full jpm test, bench at parity with main back-to-back
(4521ms vs 4619ms TOTAL under identical load).
Pins down what a transient is in Jolt (tagged table over a native Janet
array/table, canonical-keyed for maps/sets), where behavior deviates from the
JVM (O(n) transient/persistent! edges with O(1) native ops between, no
owner-thread check — same as Clojure 1.7+, transient-of-list leniency), and
the three reasons the machinery is seed-resident rather than a migration
candidate: it IS the mutation kernel, it sits under the seed's own dispatch,
and the value layer is declared irreducible. Exists so the kernel-shrink
ladder (jolt-tzo) doesn't revisit transients every round.
init in compile mode is ~2.4 s (tier loading, analyzer self-compile, macro
recompilation), paid by every process that builds a ctx from source — each
jpm-test file, embedders, workers. init-cached marshals the built ctx to a
disk image (same root-env dicts as snapshot/fork) and later processes
unmarshal it in ~5 ms, any process: nothing from the baking process is
needed at load.
The cache key fingerprints the embedded .clj stdlib (which covers jolt-core:
analyzer, IR, core tiers), the .janet seed sources next to the module, the
janet version, the init opts, and the env knobs that shape a ctx (JOLT_PATH/
MUTABLE/AOT_CORE/FEATURES) — any change rebuilds. Corrupt or non-ctx images
fall back to a rebuild (unmarshal of garbage can 'succeed' with a scalar, so
the shape is checked, not just the throw). Writes are atomic (tmp + rename)
so racing cold starts never publish a torn image. JOLT_NO_IMAGE_CACHE=1
opts out; JOLT_IMAGE_CACHE_DIR overrides the location (default TMPDIR).
Test consumers switch to init-cached (harness, suite-worker, conformance,
the behavioral unit/integration tests); tests that validate the bootstrap
itself (bootstrap-fixpoint, staged-bootstrap, aot round-trip, direct-linking)
and the deps tests (tmp-dir :paths would fragment the key) keep real init.
Full jpm test: 2:46 -> 1:58 (~29%). New ctx-image-test covers cold/warm,
cross-process load (subprocess runs defn/redef/macros/protocols/multimethods
off the baked image), per-opts keying, and corrupt-image fallback.
The 1104-line Janet bootstrap compiler existed to build jolt.ir/jolt.analyzer
and the kernel tier before the self-hosted analyzer could exist. It is
replaced by the interpreter + one fixpoint turn:
1. bootstrap-load-source loads the compiler sources INTERPRETED (the
evaluator can run the analyzer — it always could).
2. After the overlay is up, self-compile-compiler! re-runs the kernel tier,
jolt.ir, and jolt.analyzer through the SELF-HOSTED pipeline — the
interpreted analyzer compiles itself, and steady state runs compiled with
no bootstrap compiler involved.
Measured: init {:compile? true} 1093 -> ~2400 ms (the one-time interpreted
pass + self-compilation), but steady-state compilation is 2.8x FASTER
(100 forms: 134 -> 48 ms) — the self-hosted pipeline emits better code than
the bootstrap did. An AOT image for init cost is future work (aot.janet's
machinery is the natural vehicle).
The bootstrap's runtime kernel moves to backend.janet (jolt-runtime-env,
ctx-janet-env, build-map-literal); aot imports it from there. The
uncompilable-error? punt check unwraps the interpreter's exception struct
(the interpreted analyzer's throw arrives wrapped). compile-string/
compile-file (the bootstrap's source-text emitter API, no callers outside
the bootstrap's own unit test) are removed with it, as is compiler-test.
Gate green across everything incl. fixpoint stage1==2==3, AOT round-trip,
uberscript, CLI; conformance 326x3; suite 4566 >= 4540; bench in band.
make-hierarchy/derive/underive/isa?/parents/ancestors/descendants are now
Clojure in the overlay — Clojure's own pure-map implementation: a hierarchy
is {:parents {tag #{..}} :ancestors {..} :descendants {..}}, the 3-arity
forms are PURE (derive returns a new hierarchy), and the 1/2-arity forms swap
a private global-hierarchy atom.
This fixes three correctness gaps the Janet kernel had: multi-parent derive
(the kernel's :parents held a single parent per tag), TRANSITIVE descendants
(the kernel tracked direct children only), and vector-pair isa?
((isa? [child1 child2] [parent1 parent2])). Cyclic and duplicate derives now
behave like Clojure (throw / no-op).
Multimethod dispatch was the kernel's only internal caller: defmulti-setup's
dispatch closure now calls the overlay's isa? through a lazily-resolved var
(cached per multimethod); a :hierarchy option is an atom (deref per dispatch,
matching Clojure's var semantics) or a plain hierarchy map. The Janet kernel
(types.janet) and the core-* wrappers are deleted.
20 new hierarchy spec rows (pure 3-arity incl. cycle/duplicate edges, global
+ dispatch incl. custom :hierarchy atoms). Gate green: conformance 326x3,
suite 4566 >= 4540, all batteries, bench in the session band.
The representation predicates (sequential?/associative?/counted?/indexed?/
reversible?/seqable?, boolean?/double?/float?/infinite?, the qualified/simple
ident predicates), the realization boundaries doall/dorun, list*, find,
realized?/force, pop, the print-str family, and rand-nth/random-sample are
now Clojure in 20-coll, expressed over the overlay's own predicates and
primitives — no Janet representations referenced.
Several got MORE correct in the move: dorun honors its bounded-n arity (the
seed ignored n); find works on vectors by index (contains? gives it free);
indexed? no longer claims seq results; seqable? no longer claims arbitrary
tagged tables (atoms were "seqable"); realized? reads the tagged slots via
the established get pattern.
Seed: 3427 -> ~3200 lines, 371 -> ~345 core-* fns. Gate green (conformance
326x3, suite 4569 >= 4540, stdlib battery, all specs+unit); bench in noise.
This project uses **bd** (beads) for issue tracking. Run `bd prime` for full workflow context.
> **Architecture in one line:** Issues live in a local Dolt database
> (`.beads/dolt/`); cross-machine sync uses `bd dolt push/pull` (a
> git-compatible protocol), stored under `refs/dolt/data` on your git
> remote — separate from `refs/heads/*` where your code lives.
> `.beads/issues.jsonl` is a passive export, not the wire protocol.
>
> See [SYNC_CONCEPTS.md](https://github.com/gastownhall/beads/blob/main/docs/SYNC_CONCEPTS.md)
> for the one-screen overview and anti-patterns (don't treat JSONL as the
> source of truth; don't `bd import` during normal operation; don't
> reach for third-party Dolt hosting before trying the default).
## Quick Reference
```bash
bd ready # Find available work
bd show <id> # View issue details
bd update <id> --claim # Claim work atomically
bd close <id> # Complete work
bd dolt push # Push beads data to remote
```
## Non-Interactive Shell Commands
**ALWAYS use non-interactive flags** with file operations to avoid hanging on confirmation prompts.
Shell commands like `cp`, `mv`, and `rm` may be aliased to include `-i` (interactive) mode on some systems, causing the agent to hang indefinitely waiting for y/n input.
**Use these forms instead:**
```bash
# Force overwrite without prompting
cp -f source dest # NOT: cp source dest
mv -f source dest # NOT: mv source dest
rm -f file # NOT: rm file
# For recursive operations
rm -rf directory # NOT: rm -r directory
cp -rf source dest # NOT: cp -r source dest
```
**Other commands that may prompt:**
- `scp` - use `-o BatchMode=yes` for non-interactive
- `ssh` - use `-o BatchMode=yes` to fail instead of prompting
- `apt-get` - use `-y` flag
- `brew` - use `HOMEBREW_NO_AUTO_UPDATE=1` env var
<!-- BEGIN BEADS INTEGRATION v:1 profile:minimal hash:7510c1e2 -->
## Beads Issue Tracker
This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands.
### Quick Reference
```bash
bd ready # Find available work
bd show <id> # View issue details
bd update <id> --claim # Claim work
bd close <id> # Complete work
```
### Rules
- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists
- Run `bd prime` for detailed command reference and session close protocol
- Use `bd remember` for persistent knowledge — do NOT use MEMORY.md files
**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/SYNC_CONCEPTS.md for details and anti-patterns.
## Session Completion
**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds.
**MANDATORY WORKFLOW:**
1. **File issues for remaining work** - Create issues for anything that needs follow-up
This file provides instructions and context for AI coding agents working on this project.
<!-- BEGIN BEADS INTEGRATION v:1 profile:minimal hash:7510c1e2 -->
## Beads Issue Tracker
This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands.
### Quick Reference
```bash
bd ready # Find available work
bd show <id> # View issue details
bd update <id> --claim # Claim work
bd close <id> # Complete work
```
### Rules
- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists
- Run `bd prime` for detailed command reference and session close protocol
- Use `bd remember` for persistent knowledge — do NOT use MEMORY.md files
**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/SYNC_CONCEPTS.md for details and anti-patterns.
## Session Completion
**When ending a work session**, you MUST complete ALL steps below. Work is NOT complete until `git push` succeeds.
**MANDATORY WORKFLOW:**
1. **File issues for remaining work** - Create issues for anything that needs follow-up
A Clojure implementation on top of [Janet](https://janet-lang.org). Jolt reads Clojure source and, by default, compiles each form to native Janet bytecode — falling back to a tree-walking interpreter for forms the compiler doesn't handle, so results always match the interpreter. It ships a Clojure-compatible standard library. The goal is a Janet-hosted [SCI](https://github.com/borkdude/sci)-style runtime with a minimal bootstrap.
A Clojure implementation on [Chez Scheme](https://cisco.github.io/ChezScheme/).
Jolt reads Clojure source, analyzes it to a host-neutral IR, emits Scheme, and
runs it on Chez. The compiler is self-hosted: it is written in Clojure
(`jolt-core/`) and compiles itself. It ships a Clojure-compatible standard library.
## Install
Grab the self-contained `joltc` binary (Linux/macOS/Windows) — it bundles the
runtime, compiler, and standard library, so there is nothing else to install.
Download the binary archive for your platform from the
For compute-heavy code the compiled path is dramatically faster than tree-walking,
at native Janet speed.
**Validated at parity.** The conformance suite passes 258/258 under *all three*
execution paths — interpreter, compiler, and the self-hosted compiler
(`conformance-test.janet` runs all three in CI) — and the full clojure-test-suite
matches its baseline across ~4.6k assertions — evidence the hybrid path doesn't
diverge.
**AOT.** `aot.janet` marshals a compiled namespace to a Janet bytecode image
(`save-ns`) and loads it back into a fresh context (`load-ns-image`), skipping
parse/analyze/emit/compile on reload. Core fns are referenced by name against the
baked-in runtime; only user bytecode and var cells are serialized.
## Host interop
Jolt exposes CLJS-style host interop through `.` on any Janet table or struct — a field holding a function is called with the receiver as the first argument:
`clojure.core.async`, runtime `eval`/`load-string`/`defmacro`, and the full
reader (`#()`, `#_`, `#?`, tagged literals, `#"…"`) all behave as on the JVM.
`=` is category-aware (`(= 3 3.0)` ⇒ `false`) and `==` is value-equality, as in
Clojure. The genuine divergences:
- **Host platform.** No JVM and no Java interop — `import`, `gen-class`, `proxy` of Java classes, and `java.*` are unavailable. `instance?` recognizes a small set of built-in types (`clojure.lang.Atom`, `Number`, `String`, …).
- **Numbers.** Janet integers and doubles. `(/ 1 3)` is `0.3333…` and large products lose precision. No ratios or `BigDecimal` (`ratio?` is always false, `bigdec` falls back to a double); `bigint`/`biginteger` use Janet's 64-bit `int/s64`, not arbitrary precision. The reader still accepts Clojure's numeric literal syntaxes — the BigInt/BigDecimal suffixes (`42N`, `1.5M`), ratios (`1/2`), radixed integers (`2r1010`, `16rFF`), and exponents (`1e3`) — but reads them as plain Janet numbers (a ratio becomes its double quotient). The auto-promoting `+'`/`-'`/`*'`/`inc'`/`dec'` are aliases for the plain ops, since Janet numbers don't overflow. `quot`/`rem`/`mod` follow Clojure's sign rules. The symbolic values `##Inf`/`##-Inf`/`##NaN` read, and `infinite?`/`NaN?` work. Janet represents an integer and an integer-valued double identically, so `1` and `1.0` are indistinguishable: `(float?/double? 1.0)` is `false` and `(int? 1.0)` is `true` — `float?`/`double?` are true only for values with a fractional part or `##Inf`/`##NaN`.
- **Collections.** By default Jolt uses immutable persistent data structures: vectors are 32-way branching tries (structural-sharing persistent vectors with O(log₃₂ n) `conj`/`assoc`/`nth`), lists are persistent singly-linked cons cells (O(1) `conj`/`cons` prepend with structural sharing), and maps/sets are persistent hash structures. Value equality and sequence operations are Clojure-compatible, but hash-map/hash-set iteration order is unspecified and differs from Clojure — use `sorted-map`/`sorted-set` when order matters.
- **Mutable build mode.** Jolt can be compiled to use fast Janet-native *mutable* collections instead, via a build-time flag: `JOLT_MUTABLE=1 jpm build` (default `jpm build` is immutable). In mutable mode vectors and lists share one mutable array representation (so `conj` mutates in place and appends, and `vector?`/`list?` no longer distinguish them) — a performance/looseness trade-off. The default immutable build has full Clojure value semantics.
- **Concurrency / STM.** No refs, `dosync`, agents, or `send`; `locking` evaluates its body without real locking. Atoms, volatiles, promises, and delays are supported.
- **Futures.**`future` runs its body on a *real* OS thread (Janet's `ev/thread`), so it can use a second core for CPU-bound work — unlike the cooperatively-scheduled `go` blocks. `deref`/`@` parks until the result is ready (with the optional `(deref f timeout-ms timeout-val)` arity); `future?`, `future-done?`, `realized?`, `future-cancel`, and `future-cancelled?` are supported. Two important divergences from the JVM: (1) **snapshot semantics** — Janet threads have separate heaps, so the body and the state it closes over are *copied* to the worker thread and only the return value is copied back; mutating a captured atom does not propagate to the parent (communicate via the return value). (2) **no thread interruption** — Janet OS threads can't be cancelled mid-run, so `future-cancel` marks the *future* cancelled (deref then throws and the predicates flip) but the underlying computation still runs to completion in the background. As on the JVM, a live future thread keeps the process alive until it finishes (the JVM's non-daemon future pool behaves the same).
- **core.async.**`clojure.core.async` runs on Janet fibers and channels (`chan`, `go`, `go-loop`, `<!`/`>!`/`<!!`/`>!!`, `close!`, `alts!`, `timeout`, `put!`/`take!`, `buffer`/`dropping-buffer`/`sliding-buffer`, and channel transducers via `(chan n xform)`). Because Janet fibers are stackful coroutines, a `go` block is just its body run in a fiber — no CPS/state-machine rewrite — so `<!`/`>!` work *anywhere*, including inside `try`, nested `fn`s, and loops (positions Clojure's `go` macro forbids). Go blocks are cooperatively scheduled on one OS thread, so parking (`<!`) and blocking (`<!!`) coincide; `thread` runs cooperatively too. Dynamic-var bindings are conveyed into `go` blocks (each go block sees the bindings in effect when it was spawned).
- **Regex.** Compiled to Janet's PEG engine (Janet has no regex). Supported: capturing groups (`[whole g1 …]`), greedy and lazy quantifiers with backtracking, `(?:…)`, lookahead `(?=…)`/`(?!…)`, alternation, anchors `^ $ \b \B`, character classes, and the `(?i)` flag. Not supported: lookbehind, backreferences (`\1`), named groups (`(?<name>…)`), and Unicode property classes (`\p{Lu}`).
- **Arrays.** Java-style arrays map onto Janet's native types: `byte-array` is a Janet buffer (contiguous, C-backed); `object-array`/`int-array`/`double-array`/etc. are Janet arrays. `aget`/`aset`/`alength`/`aclone` work over both.
- **Transients.**`transient`/`conj!`/`assoc!`/`dissoc!`/`disj!`/`pop!`/`persistent!` are real mutable scratch collections backed by Janet's native arrays and tables (vectors → arrays, maps/sets → tables), so building a collection with them avoids the per-step copying of the persistent path (notably for maps/sets). `persistent!` freezes back to a persistent value.
- **Not implemented.** JVM reflection, `proxy`, and the `clojure.repl`/`clojure.template` namespaces.
Supported and Clojure-compatible: chars as a distinct type, lazy/infinite sequences, transducers, destructuring, multimethods with hierarchies, protocols/records (`deftype`/`defrecord`/`reify`/`extend-protocol`), metadata, namespaces, and the reader (`#()`, `#_`, `#?`, tagged literals, `#"…"`).
- **No JVM, no Java interop.** No reflection, no `gen-class`/`proxy`. Interop
syntax (`Class.`, `Class/static`, `.method`) resolves only against a shimmed
subset of the `java.*` standard library; a class token is a name, not a loaded
class. See [docs/host-interop.md](docs/host-interop.md). To call C libraries
directly, use the `jolt.ffi` foreign-function interface (how the db and
http-client libraries bind SQLite/libpq and sockets/OpenSSL/zlib).
- **No `BigDecimal`.**`decimal?` is always false and there is no `M` literal;
the rest of the numeric tower matches the JVM.
- **No STM.** No `ref`/`dosync`/`alter`/`commute` — coordinated shared state uses
atoms (per-atom mutex, JVM-style CAS). The concurrency primitives above are
otherwise present and run on a shared heap.
- **Regex engine.** Patterns compile through
[irregex](https://github.com/ashinn/irregex) (vendored), not
`java.util.regex`; common patterns work, Java-specific features can differ.
- **Coverage.**`clojure.core` is implemented function by function against the
JVM-sourced conformance corpus — broad but not total; a namespace can load with
most functions working and a few not yet implemented.
## Test
```
jpm test # full suite (recurses test/)
janet test/spec/sequences-spec.janet # a single spec
janet test/integration/conformance-test.janet
```bash
make test # the full gate
make corpus # conformance corpus vs the JVM-sourced spec
make unit # host-specific unit cases
make selfhost # bootstrap fixpoint (rebuild == checked-in seed)
make smoke # bin/joltc CLI smoke
make sci # load borkdude/sci's source through joltc (compat stress)
make ffi # HTTP-server GC-safety + http-client temp paths
make transient # transient mutation + linear-time builds
make certify # JVM oracle (skips if clojure is absent)
```
Tests are organized in three layers:
- **`test/spec/`** — the contract. Black-box, behavior-defining tables (one file
per public API area) that collectively pin down Jolt's defined behavior. This
is the authoritative description of what Jolt promises.
- **`test/integration/`** — cross-cutting and regression batteries: the Clojure
conformance suite (run in all three execution modes), SCI bootstrap/runtime
loading, jank conformance, the cross-dialect
[clojure-test-suite](https://github.com/jank-lang/clojure-test-suite) (a git
submodule at `vendor/clojure-test-suite`, run via a minimal `clojure.test` shim
and baseline-guarded), compile-mode tests, the library API, and a broad
systematic-coverage net.
- **`test/unit/`** — white-box tests for individual components (reader,
Where things live and what to read before changing them. Start here to answer
"where does feature X live?" and "what else do I need to touch?"
## Areas
| Area | Directory | Responsibility | Re-mint? |
| --- | --- | --- | --- |
| Chez runtime | `host/chez/*.ss` | The substrate: value model, persistent collections, seqs, vars/namespaces, host interop, native `clojure.core` shims, regex, FFI, IO, the **reader**. Composed by `rt.ss`. | only `reader.ss` |
| Compiler | `jolt-core/jolt/*.clj` | analyzer → IR → backend, the optimization passes, the CLI, the deps resolver, nREPL. Baked into the seed. | **yes** |
| `clojure.core` overlay | `jolt-core/clojure/core/NN-*.clj` | Portable `clojure.core` in dependency-ordered tiers (`00-syntax` … `50-io`); the `NN` prefix *is* the load order. | **yes** |
| Tests & gate | `test/chez/`, `test/conformance/`, `host/chez/run-*.ss`, `Makefile` | Corpus (JVM oracle), unit, per-feature tests. Every `make` target has a comment. | no |
**The reader is in `host/chez/reader.ss`** (Scheme, a seed source) — *not* in
`jolt-core/jolt/` with the rest of the compiler. Re-mint applies to it.
`rt.ss` is the runtime's load-order manifest: it `(load …)`s every shim in
dependency order with a per-file comment. Read it to see how the runtime is
Design notes for non-obvious language and compiler decisions. An RFC records *why*
a thing is built the way it is; the code is the source of truth for *how*.
| # | Title | Status | Governs |
| --- | --- | --- | --- |
| [0001](0001-language-specification.md) | A Specification for the Clojure Language | Draft | The conformance target — what "is Clojure" means for jolt. |
| `loop*` | recur arity must match bindings; recur rebinds in place |
| `letfn` | mutually-recursive local fns (`letrec*` semantics — a fn body sees every binding, not only earlier ones). jolt treats `letfn` as a primitive special, not the reference's `letfn` macro → `letfn*` indirection; behavior is identical |
| `recur` | tail-position rule (normative definition of tail position needed), across `if`/`do`/`let*`/`try` interactions |
| `quote` | self-evaluation table: which literals are self-evaluating unquoted |
| `var` | `#'` reader sugar; resolution at compile time |
| `throw` | any value vs Throwable — host question; jolt/cljs allow data, reference requires Throwable → classification needed |
| `try/catch/finally` | catch dispatch order, `:default`-style catch-all is a dialect extension (⚠ divergence note), finally evaluation guarantees, value of try |
| `set!` | host-dependent (dynamic vars + host fields) |
| `set!` | three targets, all implemented: `(set! *var* val)` sets the var's innermost thread binding (else root); `(set! field val)` inside a `deftype` method mutates a `^:unsynchronized-mutable`/`^:volatile-mutable` field in place; `(set! (.-field obj) val)` does the same via interop syntax. Returns val |
