irregex builds a POSIX leftmost-longest DFA for a pattern when it can, and jolt
used it for everything. For a pattern with an alternation whose branches have
capturing groups, that DFA leaks a non-participating branch's group: e.g.
#"(?:([0-9])|([0-9])r([0-9]+))" on "2r11" left group 1 = "2" instead of nil, so
tools.reader (rewrite-clj's dep) misread 2r1100 as 2 and 16rFF as 16.
java.util.regex is itself a leftmost-first backtracking engine, so compile a
capturing pattern with irregex's backtracking matcher ('backtrack): its submatch
semantics match the JVM and it clears a losing branch's group. Non-capturing
patterns keep the DFA — with no groups to read, its whole-match result is all a
caller sees, and it avoids backtracking's worst case. The submatch count comes
from a first cheap compile; a capturing pattern recompiles once and caches.
This clears the last rewrite-clj parser-test failure (now 772/0/0). Corpus rows
for the alternation-group case and the radix read. make test green.
Running the rewrite-clj test suite under jolt exposed six bugs, each fixed here:
- `for`/`doseq` `:let` bindings never went through `destructure`, so a
destructuring pattern (`:let [{:keys [y]} x]`) hit `let*` raw and failed to
compile. Emit `let`, like Clojure.
- `with-open` couldn't close a deftype/defrecord that implements a `close` method
(java.io.Closeable / AutoCloseable, e.g. tools.reader's readers) — `__close`
only knew jhost readers and map `:close` fns. Dispatch a record's `close`.
- A deftype/defrecord method param named like a field didn't shadow the field
(the field's let-binding wrapped the params). Params now shadow, as in Clojure.
- A deftype whose simple name collided with a built-in host class clobbered it in
the global ctor table, so `(java.io.PushbackReader. …)` built tools.reader's
same-named deftype. Register deftypes/built-ins by FQN, don't let a deftype
overwrite a built-in's simple name, and qualify a bare `(Name. …)` to the
deftype's FQN only in the ns that defined it.
- `clojure.walk` was lazy over a non-list seq (missing `doall`), so a walk whose
fn has side effects read stale state. Make it eager, like Clojure.
- `Character/isWhitespace` used an ASCII-only check that missed U+2028 and other
Unicode whitespace. Use the JVM's Unicode set (minus the no-break spaces it
excludes).
Regressions: corpus rows (for-let destructure, method-param shadow, walk eager,
isWhitespace), a unit row (with-open closes a record), and smoke checks (the
class-name collision, run in a fresh -e process so the deftype doesn't leak).
One divergence remains unfixed: a submatch from a losing regex alternation branch
leaks when the winning branch has a quantified group (a bug in the vendored
irregex engine, not jolt) — tracked separately.
A registered #tag data reader whose fn returns a FORM (borkdude/html's #html
expands to (->Html (str …))) was rewritten to a runtime call (reader-fn 'inner),
so the returned code became a runtime list value instead of being compiled —
(str #html [:div]) rendered the code, not "<div>". Clojure applies a data reader
at read time and substitutes its result as code.
loader.ss now applies the reader at load time: a code form (a list) is spliced in
to be compiled, a value (time-literals #time/date -> a Date) keeps the runtime
call, which also keeps a non-serializable constant out of an AOT build. The build
emit path never applied data readers at all (a #tag literal failed a `jolt build`
with "unsupported form"); emit-image.ss gets an ei-emit-form-hook the build sets
to the same rewrite, left as a no-op elsewhere so the seed mint (which doesn't
load loader.ss) is unaffected and the self-host byte-fixpoint holds.
Also make clojure.test report the actual values of a failing (is (= a b)) — it
printed only the form. Restricted to the common pure predicates so a macro head
still takes the plain path.
Fixture test/chez/datareader-app + a smoke check (interpreted) and a build-smoke
check (AOT). make test green, no corpus change.
A :jolt/native spec can now carry a :static archive; `jolt build` links it
into the executable, so the app calls the C code with no shared object on the
target. --dynamic (or :jolt/build {:dynamic-natives true}) keeps the old
runtime load-shared-object behavior; a spec with no :static is unchanged.
The cc link force-loads the archive (-force_load on macOS, --whole-archive on
Linux) and exports the executable's symbols (-rdynamic on Linux) so the baked-in
symbols resolve via (load-shared-object #f) + foreign-procedure at startup. Build
step 1 evaluates the app's foreign-procedure forms in-process, so a static
archive is preloaded there as a throwaway shared object to resolve them.
The distributed self-contained joltc has no external cc/Chez but must build these
apps, so it now bundles the Chez kernel (libkernel.a + scheme.h) and the launcher
source and re-links a custom stub with the archives baked in — needing only a
system cc, no Chez. run/repl skip static-only specs (nothing to load); keep a
:darwin/:linux candidate to use such a lib interpreted.
Adds static-native-smoke (cc path) to ci and a static phase to the joltc
self-build smoke (distributed path).
repl-form-complete? entered the :regex state on '#' but only consumed the
'#', so the opening '"' was then read by the :regex handler as the CLOSING
quote. The regex body got scanned in :code state, and any delimiter or quote
inside it (a group like #"(a)", a char class #"[0-9]+") threw off the
paren/string count — so a one-line regex form was judged incomplete and the
REPL hung waiting for continuation lines. Consume the '#"' together.
Adds a self-checking predicate test (test/chez/repl-reader-test.clj, run via
joltc so jolt.main resolves) and an end-to-end regex REPL case in smoke.sh.
The :expected was a bare list "(0 :a 2)", but run-corpus evals :expected as
source, so it applied 0 as a fn -> "0 cannot be cast to IFn". Every other
list-valued :expected is self-evaluating; this one slipped in unquoted.
Vectorize it to [0 :a 2], matching what regen-corpus.clj produces.
The REPL evaluated one line at a time, so a form split across lines
(e.g. `(+` then `1 2)`) raised instead of waiting. The read loop now
accumulates lines until delimiters are balanced — skipping string,
char, regex and comment context — printing a `... ` continuation prompt
for each extra line.
Reader/runtime errors rendered as Chez's "attempt to apply
non-procedure #[chez-pmap...]" instead of their real message. Two causes:
jolt-throw raised the thrown value raw. When a throw crossed the host
`eval` boundary, Chez re-wrapped the non-condition into a compound
condition whose message extraction applies the value, losing the message
and crashing on ex-info's empty-map :data. jolt-throw now raises a
&jolt-throw condition wrapping the value; catch (lowered to `guard`),
jolt-report-uncaught and jolt-render-throwable unwrap it back via
jolt-unwrap-throw, so ex-data/ex-message and the backtrace tag survive.
Every reader/post-prelude EOF-throw site used `(empty-pmap)` (with
parens), applying the empty-map value as a procedure and crashing during
ex-info construction before jolt-throw ran. Fixed to `empty-pmap`.
Re-minted the seed; smoke 23/23, unit 574/574.
^C to a running `joltc --nrepl-server` aborted with "thread does not
own mutex" because the accept-loop thread absorbed SIGINT in its foreign
accept() call, where Chez can't run the keyboard-interrupt handler, and
run-main-pump's tight condition-wait loop wasn't interruptible anyway.
Block SIGINT in the primordial thread before starting the server so the
accept loop inherits a blocked mask, park in a single interruptible
condition-wait via the new park-until-interrupt, and run registered
shutdown hooks (newest-first, each isolated) from the keyboard-interrupt
handler before (exit 0). The stop fn now drops .nrepl-port via the new
jolt.host/delete-file seam — clojure.java.io/delete-file doesn't exist
in Jolt and silently no-ops, so .nrepl-port was never removed.
^D (EOF) exits cleanly in canonical mode but some terminals and editors
don't deliver it, leaving the user stuck. Accepting :repl/quit or :exit
as the first form of a line gives a reliable keyword exit that works
everywhere. The check parses the line with read-string rather than
checking the evaluated value, so a nested value that happens to print
as the keyword can't trigger an exit.
list?, ratio?, and rational? are the predicate-web members that are
genuinely safe to migrate: not extended at runtime, not on the compiler
emit/inference path, not reached by the kernel tier. They now live in the
overlay (clojure/core/20-coll.clj) built on the jolt.host tower/rep tests,
lowering to the same code the native shims did. Removed their native
definitions (predicates.ss) and, for ratio?/rational?, the now-redundant
post-prelude re-assertions. Also dropped the dead all-flonum overlay
ratio?/rational?/decimal? stubs.
The rest of the web stays native and is documented as such: map?/set?/
seq?/coll? are extended with sorted/record/lazy arms, decimal? is extended
by the optional bigdec module, integer?/float? are on the emit/inference
path, vector? is reached by the kernel-tier peek. jolt.host exposure is
therefore narrowed to just the tests these three consume (exact?,
rational-type?, cseq?, cseq-list?, empty-list?).
Numeric probe is byte-identical to pre-migration; list? correct across
list/vector/lazy/empty/cons/rest cases. Selfhost fixpoint holds, values/
unit/smoke/corpus green, bench flat within noise.
The clojure.core type predicates bottom out at host tests that overlay
Clojure can't reach. Expose them under jolt.host so the predicate web can
be built as pure compositions that lower to exactly these calls:
numeric tower: exact? flonum? integer-type? rational-type?
collection reps: pvec? pmap? pset? cseq? empty-list? cseq-list? lazyseq?
exact? is wrapped to be total (Chez's raw exact? errors on a non-number;
the others return #f for a non-match). lazyseq? is exposed in
lazy-bridge.ss because jolt-lazyseq? is defined there, after predicates.ss.
map?/set?/seq? are deliberately not reduced to a single rep test: they are
extended at runtime with sorted-collection/record/lazy arms, so only the
rep predicates are exposed, not those unions. Additive only (new bindings,
nothing references them yet); bench unchanged within noise.
Attempting to migrate the reader-conditional constructor to the overlay
revealed that an overlay defn returning a :jolt/type-tagged map literal
silently fails to bind during the seed mint: the guard around each
prelude form swallows the load-time error, leaving the var unbound. This
is the same reason every other tagged-value constructor (atom,
volatile!, tagged-literal) is native, so reader-conditional is
reclassified STAY-PRIMITIVE rather than a safe migration.
set was a native shim (apply jolt-hash-set (seq->list coll)). It is a
pure composition, so the Clojure version (apply hash-set (seq coll))
lowers to the same code. The compiler uses set, but only off the emit
path (the backend's bare-native-names def and type inference), so it can
live in the kernel tier: compiling that tier never calls set, and by the
time those callers run the tier is already bound.
This is distinct from boolean, which the backend calls for every :if
node on the emit path. Moving boolean even to the kernel tier deadlocks
(compiling the tier that defines boolean needs boolean), so boolean stays
native. Added a comment in predicates.ss recording that.
Re-mint converges in 3 passes and the benchmark suite is unchanged
within noise (collections 43.3 vs 43.1, binary-trees 367 vs 367, the
rest flat).
The overlay defines transduce in clojure/core/22-coll.clj as a pure
composition (xf (reduce xf init coll)), and it shadows the native
jolt-transduce by load order. The compiled overlay version is already
what gets baked into the seed, so the native binding in
natives-transduce.ss was dead weight.
transduce is not used by the self-hosted compiler and no overlay tier
before 22-coll references it, so removing the native binding is safe.
Re-minting produces a byte-identical seed, which proves the runtime is
unchanged. sequence stays native (its transformer iterator drives the
reduced box and lazy realization directly).
range, map, and filter were fully element-by-element lazy, so
(map f (range 1 50)) realized one element per first/nth where JVM
Clojure realizes a whole 32-element chunk. range is a chunked
LongRange on the JVM and map/filter are chunk-preserving, so the
observable side-effect timing differed.
Following clojure.lang.LongRange, ChunkedCons, ChunkBuffer and
core.clj, this adds a crest field to the cseq record and a
cseq-chunked constructor modeling ChunkedCons (a standalone chunk
pvec, an offset, and the after-chunk seq). The chunk accessors move
to seq.ss next to the representation they read. map/filter/remove
take a chunked branch when the source is chunked, realizing the whole
chunk and chunk-cons'ing it onto a lazy rest, so their output is
itself chunked and chained transforms each batch by 32. Bounded range
is now an eager chunked seq, and the reduce fast path flows through a
ChunkedCons rest. The chunk-buffer/chunk/chunk-cons builder API in
natives-array.ss now produces a real ChunkedCons.
Single-arg (range), multi-coll map, and plain lazy seqs stay
element-by-element, like the JVM.
Adds a lazy / chunking suite to the corpus that observes realization
timing via an atom counter: first over a chunked map realizes 32,
crossing a chunk boundary realizes 49, chained maps batch [32 32],
filter applies the predicate to the whole first block, and a plain
lazy seq still realizes one element at a time. Two cases that
documented the old over-laziness now assert the JVM value of 32 and
were dropped from the allowlist. certify against JVM Clojure 1.12.3
reports 0 new and 0 stale divergences.
On a pushed v* tag, build the self-contained joltc (make joltc-release) for
x86_64-linux, x86_64-macos, and aarch64-macos, package each as a tar.gz plus a
SHA256, and attach them to the GitHub Release. Linux builds Chez from source like
tests.yml (the apt package lacks the kernel dev files build-joltc cc-links
against); macOS uses Homebrew chezscheme, which ships chez and the csv kernel
files. No notarization, matching dirge — macOS tarball users de-quarantine once
or install via a Homebrew tap.
The Homebrew tap update job is a separate follow-on; this covers building and
publishing the release assets.
host/chez/build-joltc.ss builds joltc into target/<profile>/joltc: it emits a
flat source of the full runtime + compiler image + inlined build.ss + every
jolt-core/stdlib file as a baked string literal + a cli.ss-style launcher, then
(in a fresh Chez, so the inlined runtime's redefinition of error doesn't strand
early references and runtime eval still sees the runtime's top-level procedures)
compiles it and cc-links it with the Chez petite/scheme boots and the launcher
stub embedded as C arrays. The launcher reads those arrays via FFI on
(jolt-materialize-bundles!) and registers them so build-self-contained can spill
them. joltc itself is cc-linked (clean signature for Homebrew); only the apps it
later builds use the appended-stub path.
build.ss: skip the csv toolchain check on the self-contained path and create the
build dir with a subprocess-free bld-mkdir-p, so a from the
distributed binary shells out to nothing.
release = optimize-level 3 + no inspector info + compressed; debug =
optimize-level 0 + inspector + procedure source + debug-on-exception.
joltc-selfbuild-smoke.sh (make joltcsmoke) builds joltc and, with an empty
environment (no chez/cc/PATH), drives it through the build-app fixture, asserting
the produced binary's output. .gitignore ignores target/.
Adds the pieces a toolchain-free joltc needs to compile apps with no external
Chez or cc:
- host/chez/java/io.ss: register-embedded-bytes!/jolt-embedded-bytes,
read-file-bytes, jolt-spill-embedded!, jolt-append-payload! (frames an app
boot onto the stub as [stub][boot][len:le64]["JOLTBOOT"]), and jolt-chmod-755
via load-shared-object #f (no subprocess).
- host/chez/stub/launcher.c: a native stub that locates its own executable,
reads the trailing frame, and hands the appended boot to the Chez kernel.
- host/chez/loader.ss: resolve-on-roots consults the embedded source store before
disk; ldr-read-source reads baked source. Dev (empty store) is unaffected.
- host/chez/build.ss: build-binary step 4 splits into build-self-contained
(in-process compile-file/make-boot-file with the system error restored, then
append the boot to a copy of the embedded stub) and build-with-cc (the existing
dev path). The self-contained path is taken only when the stub is embedded.
The legacy cc path is unchanged behaviorally; make buildsmoke still passes.
The overlay comment claimed Jolt has no chunked seqs and that chunked-seq?
is always false. That is no longer true: a vector's seq is a real chunked
seq, and post-prelude.ss rebinds chunked-seq? to na-chunked-seq?, which
returns true for a vector seq. The defn here is only a placeholder so
references compile during overlay load. Updated the comment to say so.
Two pre-existing issues in the nrepl command, exposed when #269 moved the accept
loop into a future.
jolt.nrepl/start was invoked inside (future ...), so binding the socket happened
on a background thread. A bind failure such as the port already being in use was
captured into a future that nothing derefs and silently swallowed, leaving the
main thread parked in run-main-pump forever with no server and no error. start
now binds the socket synchronously before returning, so that failure propagates
to the caller and the process exits with a visible message. Only the blocking
accept loop runs on a worker thread.
There was also no shutdown path: the accept loop ran forever and the listen
socket was never closed. start now returns a stop fn that breaks the accept
loop, closes the socket to free the port, and removes .nrepl-port. main.clj runs
run-main-pump and then calls the stop fn, so a stop-main-pump (from a glimmer app
on quit, or from nrepl-evaluated code) shuts the server down cleanly and the
process exits.
PR #269 added the main-thread executor (call-on-main-thread, run-main-pump,
stop-main-pump) so the nREPL accept loop can run on a worker thread while the
primordial thread owns the GUI main loop. Two problems made stop-main-pump
unusable as a graceful-shutdown or external API.
run-main-pump set jolt-main-pump-active to #t on entry but never cleared it on
exit, so after the pump returned the flag stayed #t. call-on-main-thread also
read that flag outside the queue mutex, so even with a reset there was a window
where a job could be enqueued just as the pump left, then block forever on a
pump that was gone.
Both are now decided under jolt-main-queue-mu. The pump clears active in the
same critical section where it sees the stop flag and an empty queue, and
call-on-main-thread reads active and enqueues atomically under that lock. A
caller that loses the race sees the pump inactive and runs the thunk inline,
the same fallback used when no pump is running, rather than blocking. A
dynamic-wind around the loop also clears active on an abnormal exit so a later
run-main-pump starts clean.
Match babashka's spelling: the nREPL server now starts with
`bin/joltc --nrepl-server [port]` instead of `bin/joltc nrepl`. Port
parsing and JOLT_NREPL_PORT are unchanged.
Also wire up --help/-h to print usage (previously only the no-arg
invocation did), and fix the usage listing to show the real flag.
Smoke now asserts --help mentions --nrepl-server. Docs updated to match.
The AOT suite doesn't cover 64-bit integer arithmetic (Chez fixnums are 61-bit,
so genuine 64-bit values are bignums) — the SplitMix PRNG behind test.check is
the worst case. Add the measured jolt-vs-JVM numbers for the PRNG/mix-64 and the
generator workload: the bitwise native-ops + var-cell caching took mix-64 from
~18x to ~3.2x JVM and the PRNG from ~30x to ~12x; the residual is the open-world
generator dispatch/allocation and the bignum floor, not arithmetic.
Profiling jolt-i5if showed <=60-bit arithmetic is already native-fast; the real
general overhead in the run/-e/-m path is var resolution. Every var reference
compiled to (var-deref ns name), which builds + hashes a fresh "ns/name" string
and does a hashtable lookup per access (~45ns). The var cell is interned and
def-var! mutates it in place, so caching the resolved cell is sound under
redefinition.
Generalize the devirt per-site cache-cell mechanism to var value references: a
ref inside a fn resolves its cell once into the def's closure, then reads it via
var-cell-deref (a field read after the first). var-cell-deref is the cell-based
var-deref — binding-aware (dynamic vars + *ns* still resolve) and lenient on an
unbound root (a forward-declared var doesn't throw, unlike jolt-var-get).
Gated by a runtime flag: ON for runtime-compiled code (compile-eval.ss), OFF for
the seed mint and AOT build (emit-image.ss) so the seed stays a byte-fixpoint --
prelude.ss is unchanged, only image.ss picks up the new backend. ~5x on a
var-ref-heavy loop (1058ms->205ms); ~1.2x on test.check (its generators are more
deftype/dispatch-bound than var-deref-bound). No C/FFI.
Corpus rows pin redefinition / dynamic binding / forward ref through a cached
ref. make test + shakesmoke green, selfhost holds, SCI 211/218, certify 0-new.
Profiling the test.check distribution/large-sample slowness (jolt-i5if): the
hot path is the SplitMix PRNG, dominated by 64-bit mix arithmetic, and the
bitwise ops (bit-and/or/xor/not, shifts) were NOT in the backend native-ops
table — so (bit-xor a b) compiled to a var-deref through the variadic overlay
(__bit-xor) instead of a direct call, the way +/-/* already emit.
Map bit-and/or/xor/not to the Chez bitwise-and/ior/xor/not primitives (inlined
to native code; a non-integer operand now errors like the JVM instead of being
silently truncated) and the shifts to a direct helper call. bit-and-not stays on
its overlay — its only Scheme impl is 2-arg, so a value-position arity-3 use
would mis-emit.
mix-64 arithmetic 2.7x faster, raw split+rand-long 2.4x, gen/vector ~1.4x. The
remaining gap is the bignum-vs-native-long floor (~20x, substrate) plus the
generator machinery (deftype/fn dispatch, separate). Corpus rows added for value
position, bit-not, apply, and a full-64-bit unsigned shift.
jolt's `is` was a fixed macro with no assert-expr multimethod, and the runner
bypassed the report multimethod, so libraries couldn't register custom
assertions or custom report types (e.g. test.check's ::trial/::shrunk).
Add assert-expr (2-arg [msg form], dispatch on the form's first symbol /
:default / :always-fail), do-report routing through report, and report
:pass/:fail/:error methods that feed the counters. `is` dispatches to an
explicitly-registered assert-expr method before its inline path, so thrown?/
thrown-with-msg?/= and every built-in form stay byte-identical.
Runtime stdlib only, no re-mint. test/chez/clojure-test.clj self-checks the
extension points + full is/are/testing/thrown?/use-fixtures surface; smoke gate
runs it.
jolt unifies every integer as one exact-integer type, so (byte/short/int n)
report Long not Byte/Short/Integer and instance? Byte is false. Confirmed
substrate-inherent: (byte 5) is a Chez immediate identical? to 5 (nothing to
tag, numbers carry no metadata), and arithmetic compiles to a raw Chez + that a
boxed narrow type would crash. Value/arithmetic/equality are correct.
Certify the value-correctness (= to plain int, arithmetic promotes, is a Number)
and pin the class/instance? divergence under a new :integer-box-model category.
Data/doc only.
A 62-case jolt-vs-JVM probe across seq type identity, chunking
granularity, eagerness, and realization timing. Findings: the whole
producer family is lazy at construction (no eager bugs remain), and the
26 divergences fall into two classes that diverge by representation, not
value.
Lock in the laziness contract as certified corpus rows: construction=0
for keep/keep-indexed/map-indexed/distinct/partition-by/partition-all/
interpose/interleave/take-nth/reductions/tree-seq/replace, sequence
realizes 1, next realizes 2, rest realizes 1.
Pin the two accepted divergence classes (allowlisted, gate-guarded):
- seq-type-model: jolt reifies seqs as PersistentList/LazySeq vs JVM's
Cons/Iterate/LongRange/Repeat/Cycle/ChunkedSeq/StringSeq/KeySeq/RSeq/
ArraySeq/SubVector (jolt-aei7)
- chunking-model: unchunked, realizes one where JVM realizes a 32-chunk;
mapcat/dedupe fully lazy at construction (jolt-mm6v)
known-divergences.edn gains both categories; SPEC.md documents the seq
semantics contract. Data/doc only, no re-mint. certify 0 new / 0 stale.
A pvec is a 32-way trie, but na-chunk-first built each block by calling
pvec-v on the full backing vector — materializing all n elements to a
flat Scheme vector — then copying the 32-wide window out of it. That made
chunk-first O(n), so walking a vector chunk-by-chunk (Clojure's real
chunked map/filter fast path) was O(n^2): a ported chunked map over 500K
elements took 39s, superlinear to ~700s at 2M.
na-chunk-size equals pv-width and blocks are 32-aligned, so a block is
exactly one trie leaf — pv-chunk-for hands it back in O(log n). Copy that
leaf directly; fall back to per-index reads for the rare window that
crosses a leaf boundary. Chunked map is now linear, ~133x faster at 500K
(293ms) and within ~2.3x of the native seq loop, which makes a
clojure-in-clojure seq tier viable.
Corpus rows pin chunk-first window contents + chunk-rest boundaries
against JVM; fixed a stale 'always false' chunked-seq? label.
