`joltc build` inlines the runtime (host/chez/rt.ss and everything it loads, the
seed, compile-eval, loader, ffi, the vendored irregex) into each app binary by
reading those files off disk. That works from a jolt checkout but not from the
installed self-contained binary, which has no source tree:
joltc build -m app.core
=> Exception in call-with-input-file: failed for host/chez/rt.ss: no such file
build-joltc now bakes the exact transitive closure of files the build inlines
into the binary as embedded resources (keyed by the path the `(load "…")` forms
use), and build.ss/dce.ss read runtime source through bld-source-string, which
takes the embedded copy when present and falls back to disk otherwise. So the
same joltc builds apps both from a checkout and standalone.
The release workflow now smoke-tests a self-contained build (compile a tiny app
from an isolated dir, run it) — this is exactly what shipped broken, so it now
gates the release. buildsmoke/shakesmoke/staticnativesmoke unchanged and green.
Build tooling only — no re-mint, no runtime change.
install (root) downloads the self-contained joltc release asset for the host
platform, verifies its sha256, and drops the binary in /usr/local/bin (--dir /
--version override). Resolves the latest release via the GitHub API, clears the
macOS quarantine flag, and backs up an existing joltc. Modeled on babashka's
installer. README gets a one-line curl|bash install.
Document the register-class-supers! seam next to the other host-class hooks:
when a library class belongs to a hierarchy (a custom exception caught as
IOException, a value matching instance? across its supertypes and dispatching a
protocol extended to any of them), declare its supers once and instance?/isa?/
supers/extend-protocol all derive.
An unknown reader tag produced the reader's internal form
{:jolt/type :jolt/tagged :tag :#foo :form bar}, which tagged-literal? didn't
recognize and which leaked as a raw map when printed:
(tagged-literal? (read-string "#foo bar")) => false ; want true
(pr-str (quote [#foo bar])) => "[{:jolt/type :jolt/tagged ...}]"
Both the data path (rdr-construct-tag) and the compile path (emit-quoted) now
build a real tagged-literal for a tag with no registered reader, like Clojure's
*default-data-reader-fn*, so tagged-literal? / :tag / :form / printing all work.
clojure.edn reads raw forms through a separate __read-form-raw path and applies
:readers/:default itself, so it is unaffected.
Re-mint (backend + reader are seed sources); prelude byte-identical, image only.
make test green (selfhost holds, 0 new/stale), +2 unit rows.
jolt's own throw sites raised untyped Chez conditions with the class name buried
in an English message, so (class e) reported the opaque :object and only a broad
catch worked:
(class (try (Long/parseLong "xyz") (catch Throwable e e))) => :object
; JVM: java.lang.NumberFormatException
Raise typed throwables (jolt-host-throwable) at the Long/Double parse and
StringTokenizer sites so (class e) / .getMessage / a specific catch all reflect
the real class. And fold the exception supertype table (exception-parent) into
the one class graph: exception-isa? now resolves the simple name to its graph key
and asks jch-isa?, so exceptions and every other class share a single hierarchy.
Runtime only, no re-mint. make test green (0 new/stale), +2 corpus rows.
The "does value V declare method M; if so call it" decision was re-derived in a
dozen places with two different lookup helpers — records.ss jrec-cl and
collections.ss rec-coll-method were a byte-for-byte duplicate — and reduce only
honored a reify's own reduce method, not a deftype's:
(reduce + 100 (->Rng 5)) ; Rng deftype implementing IReduceInit
=> "not seqable" ; JVM: 110
Add iface-method / iface-call: one lookup that resolves a method for a deftype OR
a reify, with arity, and is the seam a core fn's interface arm collapses to.
jrec-cl now aliases rec-coll-method (the duplicate is gone). reduce routes its
IReduceInit arm through iface-method, so a deftype's reduce drives the reduction
like a reify's, reduced short-circuit included.
Runtime only, no re-mint. make test green (0 new/stale), +2 corpus rows.
record-method-dispatch was rebound with (set! record-method-dispatch ...) in six
files, each wrapping the previous binding, so precedence was whatever the rt.ss
load order happened to be — the true outermost arm was inst-time's Date arm, not
the one you'd guess. A type-gated wrapper that only whitelists its own methods
then errored on everything else, stealing universal Object methods from the arms
beneath it: (.getClass (java.util.Date.)) threw "No method getClass on Date",
same for File, while (class ...) and (.getClass "s") worked.
Replace the wrapper stack with an ordered list of arms (register-method-arm!,
ascending priority), each returning 'pass to defer. getClass is now one arm at
the top reached by every value, so it can't be shadowed; the three duplicate
getClass checks (dot-forms, host-static, base) collapse into it. Each former
wrapper is an arm at an explicit priority instead of an implicit load-order slot.
A library can register its own arm rather than set!-wrapping the dispatcher.
Runtime only, no re-mint. make test green (0 new/stale divergences), +1 corpus
row for getClass on Date/File.
instance?, extend-protocol dispatch, isa?/supers/ancestors, and the exception
hierarchy each read their own hand-kept table, and those tables had drifted:
(instance? clojure.lang.Associative [1 2]) was true but a protocol extended to
Associative wouldn't dispatch to a vector; keyword/IFn and seq/Seqable had the
same split; (isa? ExceptionInfo RuntimeException) was false and
(supers NumberFormatException) was empty.
Add one FQN -> direct-supers graph (class-hierarchy.ss) and derive the views
from it. value-host-tags builds on the graph closure so a vector reports
Associative/Indexed/ILookup/Counted/Seqable, a keyword reports IFn, a seq
reports Seqable/List/Counted, etc. instance? now tests membership in that same
list, so it can't disagree with dispatch. canonical-host-tag recognizes any
modeled class (was a separate literal set missing Seqable/ILookup/...).
class-direct-supers unions the graph edges and class-supers returns the
transitive closure, so the exception hierarchy answers isa?/supers/ancestors.
The graph is open: jolt.host/register-class-supers! lets a library graft its
own classes on and get every view for free.
Runtime only, no re-mint. make test green (0 new/stale divergences), +3
JVM-certified corpus rows.
joltc failed to start on Windows — "Exception in foreign-procedure: no entry for
pthread_sigmask". concurrency.ss resolves pthread_sigmask/sigemptyset/sigaddset at
load with a top-level (foreign-procedure …), which resolves its symbol eagerly;
those POSIX signal fns don't exist on Windows, so the whole runtime aborted.
Guard the three resolutions (like sched_yield/chmod already are) so a non-POSIX
host yields #f, and make jolt-set-sigint-blocked a no-op when they're unavailable.
The per-thread SIGINT mask is a POSIX-only optimization for the nREPL accept loop;
Windows delivers ^C through the console, and park-until-interrupt still parks on a
condition variable. macOS/Linux resolve the symbols as before — unchanged.
Running the whole rewrite-clj test suite (159 tests) surfaced seven more bugs;
with these it passes 3377/0/0. Each is a general jolt/JVM divergence:
- *out* was pinned to the startup stdout port, so (.write *out* …) escaped a
with-out-str capture (z/print writes via *out*). It now resolves the live
current-output-port, like print/__write, so a redirect is seen.
- nth / assoc past the end of a vector or seq threw a bare Chez error (class
:object). Throw IndexOutOfBoundsException, matching the JVM.
- A number's .toString(radix) ignored the base. Render in the base, lowercase
(rewrite-clj rebuilds 0xff / 0377 / 2r1001 through it).
- A required namespace's own :as aliases leaked into its requirer: the loaded ns
form compiles while (chez-current-ns) is still the requirer, so ce-scan-requires!
registered the loaded ns's aliases under the wrong ns and clobbered a same-named
alias there. Register an (ns NAME …) form's aliases under NAME.
- A quoted collection dropped its metadata; now it keeps USER metadata (drops the
reader's :line/:column/:file), like a Clojure quoted constant.
- enumeration-seq only did (seq e); it now drives a java.util.Enumeration through
hasMoreElements/nextElement, and StringTokenizer implements them.
Regressions: corpus rows (with-out-str/*out*, nth/assoc bounds, toString radix,
quote metadata, enumeration-seq) certified against JVM; a smoke fixture for the
alias leak (a required ns's alias must not leak). tools.reader + rewrite-clj added
to docs/libraries.md. make test green.
irregex builds a POSIX leftmost-longest DFA for a pattern when it can, and jolt
used it for everything. For a pattern with an alternation whose branches have
capturing groups, that DFA leaks a non-participating branch's group: e.g.
#"(?:([0-9])|([0-9])r([0-9]+))" on "2r11" left group 1 = "2" instead of nil, so
tools.reader (rewrite-clj's dep) misread 2r1100 as 2 and 16rFF as 16.
java.util.regex is itself a leftmost-first backtracking engine, so compile a
capturing pattern with irregex's backtracking matcher ('backtrack): its submatch
semantics match the JVM and it clears a losing branch's group. Non-capturing
patterns keep the DFA — with no groups to read, its whole-match result is all a
caller sees, and it avoids backtracking's worst case. The submatch count comes
from a first cheap compile; a capturing pattern recompiles once and caches.
This clears the last rewrite-clj parser-test failure (now 772/0/0). Corpus rows
for the alternation-group case and the radix read. make test green.
Running the rewrite-clj test suite under jolt exposed six bugs, each fixed here:
- `for`/`doseq` `:let` bindings never went through `destructure`, so a
destructuring pattern (`:let [{:keys [y]} x]`) hit `let*` raw and failed to
compile. Emit `let`, like Clojure.
- `with-open` couldn't close a deftype/defrecord that implements a `close` method
(java.io.Closeable / AutoCloseable, e.g. tools.reader's readers) — `__close`
only knew jhost readers and map `:close` fns. Dispatch a record's `close`.
- A deftype/defrecord method param named like a field didn't shadow the field
(the field's let-binding wrapped the params). Params now shadow, as in Clojure.
- A deftype whose simple name collided with a built-in host class clobbered it in
the global ctor table, so `(java.io.PushbackReader. …)` built tools.reader's
same-named deftype. Register deftypes/built-ins by FQN, don't let a deftype
overwrite a built-in's simple name, and qualify a bare `(Name. …)` to the
deftype's FQN only in the ns that defined it.
- `clojure.walk` was lazy over a non-list seq (missing `doall`), so a walk whose
fn has side effects read stale state. Make it eager, like Clojure.
- `Character/isWhitespace` used an ASCII-only check that missed U+2028 and other
Unicode whitespace. Use the JVM's Unicode set (minus the no-break spaces it
excludes).
Regressions: corpus rows (for-let destructure, method-param shadow, walk eager,
isWhitespace), a unit row (with-open closes a record), and smoke checks (the
class-name collision, run in a fresh -e process so the deftype doesn't leak).
One divergence remains unfixed: a submatch from a losing regex alternation branch
leaks when the winning branch has a quantified group (a bug in the vendored
irregex engine, not jolt) — tracked separately.
A registered #tag data reader whose fn returns a FORM (borkdude/html's #html
expands to (->Html (str …))) was rewritten to a runtime call (reader-fn 'inner),
so the returned code became a runtime list value instead of being compiled —
(str #html [:div]) rendered the code, not "<div>". Clojure applies a data reader
at read time and substitutes its result as code.
loader.ss now applies the reader at load time: a code form (a list) is spliced in
to be compiled, a value (time-literals #time/date -> a Date) keeps the runtime
call, which also keeps a non-serializable constant out of an AOT build. The build
emit path never applied data readers at all (a #tag literal failed a `jolt build`
with "unsupported form"); emit-image.ss gets an ei-emit-form-hook the build sets
to the same rewrite, left as a no-op elsewhere so the seed mint (which doesn't
load loader.ss) is unaffected and the self-host byte-fixpoint holds.
Also make clojure.test report the actual values of a failing (is (= a b)) — it
printed only the form. Restricted to the common pure predicates so a macro head
still takes the plain path.
Fixture test/chez/datareader-app + a smoke check (interpreted) and a build-smoke
check (AOT). make test green, no corpus change.
A :jolt/native spec can now carry a :static archive; `jolt build` links it
into the executable, so the app calls the C code with no shared object on the
target. --dynamic (or :jolt/build {:dynamic-natives true}) keeps the old
runtime load-shared-object behavior; a spec with no :static is unchanged.
The cc link force-loads the archive (-force_load on macOS, --whole-archive on
Linux) and exports the executable's symbols (-rdynamic on Linux) so the baked-in
symbols resolve via (load-shared-object #f) + foreign-procedure at startup. Build
step 1 evaluates the app's foreign-procedure forms in-process, so a static
archive is preloaded there as a throwaway shared object to resolve them.
The distributed self-contained joltc has no external cc/Chez but must build these
apps, so it now bundles the Chez kernel (libkernel.a + scheme.h) and the launcher
source and re-links a custom stub with the archives baked in — needing only a
system cc, no Chez. run/repl skip static-only specs (nothing to load); keep a
:darwin/:linux candidate to use such a lib interpreted.
Adds static-native-smoke (cc path) to ci and a static phase to the joltc
self-build smoke (distributed path).
repl-form-complete? entered the :regex state on '#' but only consumed the
'#', so the opening '"' was then read by the :regex handler as the CLOSING
quote. The regex body got scanned in :code state, and any delimiter or quote
inside it (a group like #"(a)", a char class #"[0-9]+") threw off the
paren/string count — so a one-line regex form was judged incomplete and the
REPL hung waiting for continuation lines. Consume the '#"' together.
Adds a self-checking predicate test (test/chez/repl-reader-test.clj, run via
joltc so jolt.main resolves) and an end-to-end regex REPL case in smoke.sh.
The :expected was a bare list "(0 :a 2)", but run-corpus evals :expected as
source, so it applied 0 as a fn -> "0 cannot be cast to IFn". Every other
list-valued :expected is self-evaluating; this one slipped in unquoted.
Vectorize it to [0 :a 2], matching what regen-corpus.clj produces.
The REPL evaluated one line at a time, so a form split across lines
(e.g. `(+` then `1 2)`) raised instead of waiting. The read loop now
accumulates lines until delimiters are balanced — skipping string,
char, regex and comment context — printing a `... ` continuation prompt
for each extra line.
Reader/runtime errors rendered as Chez's "attempt to apply
non-procedure #[chez-pmap...]" instead of their real message. Two causes:
jolt-throw raised the thrown value raw. When a throw crossed the host
`eval` boundary, Chez re-wrapped the non-condition into a compound
condition whose message extraction applies the value, losing the message
and crashing on ex-info's empty-map :data. jolt-throw now raises a
&jolt-throw condition wrapping the value; catch (lowered to `guard`),
jolt-report-uncaught and jolt-render-throwable unwrap it back via
jolt-unwrap-throw, so ex-data/ex-message and the backtrace tag survive.
Every reader/post-prelude EOF-throw site used `(empty-pmap)` (with
parens), applying the empty-map value as a procedure and crashing during
ex-info construction before jolt-throw ran. Fixed to `empty-pmap`.
Re-minted the seed; smoke 23/23, unit 574/574.
^C to a running `joltc --nrepl-server` aborted with "thread does not
own mutex" because the accept-loop thread absorbed SIGINT in its foreign
accept() call, where Chez can't run the keyboard-interrupt handler, and
run-main-pump's tight condition-wait loop wasn't interruptible anyway.
Block SIGINT in the primordial thread before starting the server so the
accept loop inherits a blocked mask, park in a single interruptible
condition-wait via the new park-until-interrupt, and run registered
shutdown hooks (newest-first, each isolated) from the keyboard-interrupt
handler before (exit 0). The stop fn now drops .nrepl-port via the new
jolt.host/delete-file seam — clojure.java.io/delete-file doesn't exist
in Jolt and silently no-ops, so .nrepl-port was never removed.
^D (EOF) exits cleanly in canonical mode but some terminals and editors
don't deliver it, leaving the user stuck. Accepting :repl/quit or :exit
as the first form of a line gives a reliable keyword exit that works
everywhere. The check parses the line with read-string rather than
checking the evaluated value, so a nested value that happens to print
as the keyword can't trigger an exit.
list?, ratio?, and rational? are the predicate-web members that are
genuinely safe to migrate: not extended at runtime, not on the compiler
emit/inference path, not reached by the kernel tier. They now live in the
overlay (clojure/core/20-coll.clj) built on the jolt.host tower/rep tests,
lowering to the same code the native shims did. Removed their native
definitions (predicates.ss) and, for ratio?/rational?, the now-redundant
post-prelude re-assertions. Also dropped the dead all-flonum overlay
ratio?/rational?/decimal? stubs.
The rest of the web stays native and is documented as such: map?/set?/
seq?/coll? are extended with sorted/record/lazy arms, decimal? is extended
by the optional bigdec module, integer?/float? are on the emit/inference
path, vector? is reached by the kernel-tier peek. jolt.host exposure is
therefore narrowed to just the tests these three consume (exact?,
rational-type?, cseq?, cseq-list?, empty-list?).
Numeric probe is byte-identical to pre-migration; list? correct across
list/vector/lazy/empty/cons/rest cases. Selfhost fixpoint holds, values/
unit/smoke/corpus green, bench flat within noise.
The clojure.core type predicates bottom out at host tests that overlay
Clojure can't reach. Expose them under jolt.host so the predicate web can
be built as pure compositions that lower to exactly these calls:
numeric tower: exact? flonum? integer-type? rational-type?
collection reps: pvec? pmap? pset? cseq? empty-list? cseq-list? lazyseq?
exact? is wrapped to be total (Chez's raw exact? errors on a non-number;
the others return #f for a non-match). lazyseq? is exposed in
lazy-bridge.ss because jolt-lazyseq? is defined there, after predicates.ss.
map?/set?/seq? are deliberately not reduced to a single rep test: they are
extended at runtime with sorted-collection/record/lazy arms, so only the
rep predicates are exposed, not those unions. Additive only (new bindings,
nothing references them yet); bench unchanged within noise.
Attempting to migrate the reader-conditional constructor to the overlay
revealed that an overlay defn returning a :jolt/type-tagged map literal
silently fails to bind during the seed mint: the guard around each
prelude form swallows the load-time error, leaving the var unbound. This
is the same reason every other tagged-value constructor (atom,
volatile!, tagged-literal) is native, so reader-conditional is
reclassified STAY-PRIMITIVE rather than a safe migration.
set was a native shim (apply jolt-hash-set (seq->list coll)). It is a
pure composition, so the Clojure version (apply hash-set (seq coll))
lowers to the same code. The compiler uses set, but only off the emit
path (the backend's bare-native-names def and type inference), so it can
live in the kernel tier: compiling that tier never calls set, and by the
time those callers run the tier is already bound.
This is distinct from boolean, which the backend calls for every :if
node on the emit path. Moving boolean even to the kernel tier deadlocks
(compiling the tier that defines boolean needs boolean), so boolean stays
native. Added a comment in predicates.ss recording that.
Re-mint converges in 3 passes and the benchmark suite is unchanged
within noise (collections 43.3 vs 43.1, binary-trees 367 vs 367, the
rest flat).
The overlay defines transduce in clojure/core/22-coll.clj as a pure
composition (xf (reduce xf init coll)), and it shadows the native
jolt-transduce by load order. The compiled overlay version is already
what gets baked into the seed, so the native binding in
natives-transduce.ss was dead weight.
transduce is not used by the self-hosted compiler and no overlay tier
before 22-coll references it, so removing the native binding is safe.
Re-minting produces a byte-identical seed, which proves the runtime is
unchanged. sequence stays native (its transformer iterator drives the
reduced box and lazy realization directly).
range, map, and filter were fully element-by-element lazy, so
(map f (range 1 50)) realized one element per first/nth where JVM
Clojure realizes a whole 32-element chunk. range is a chunked
LongRange on the JVM and map/filter are chunk-preserving, so the
observable side-effect timing differed.
Following clojure.lang.LongRange, ChunkedCons, ChunkBuffer and
core.clj, this adds a crest field to the cseq record and a
cseq-chunked constructor modeling ChunkedCons (a standalone chunk
pvec, an offset, and the after-chunk seq). The chunk accessors move
to seq.ss next to the representation they read. map/filter/remove
take a chunked branch when the source is chunked, realizing the whole
chunk and chunk-cons'ing it onto a lazy rest, so their output is
itself chunked and chained transforms each batch by 32. Bounded range
is now an eager chunked seq, and the reduce fast path flows through a
ChunkedCons rest. The chunk-buffer/chunk/chunk-cons builder API in
natives-array.ss now produces a real ChunkedCons.
Single-arg (range), multi-coll map, and plain lazy seqs stay
element-by-element, like the JVM.
Adds a lazy / chunking suite to the corpus that observes realization
timing via an atom counter: first over a chunked map realizes 32,
crossing a chunk boundary realizes 49, chained maps batch [32 32],
filter applies the predicate to the whole first block, and a plain
lazy seq still realizes one element at a time. Two cases that
documented the old over-laziness now assert the JVM value of 32 and
were dropped from the allowlist. certify against JVM Clojure 1.12.3
reports 0 new and 0 stale divergences.
On a pushed v* tag, build the self-contained joltc (make joltc-release) for
x86_64-linux, x86_64-macos, and aarch64-macos, package each as a tar.gz plus a
SHA256, and attach them to the GitHub Release. Linux builds Chez from source like
tests.yml (the apt package lacks the kernel dev files build-joltc cc-links
against); macOS uses Homebrew chezscheme, which ships chez and the csv kernel
files. No notarization, matching dirge — macOS tarball users de-quarantine once
or install via a Homebrew tap.
The Homebrew tap update job is a separate follow-on; this covers building and
publishing the release assets.
host/chez/build-joltc.ss builds joltc into target/<profile>/joltc: it emits a
flat source of the full runtime + compiler image + inlined build.ss + every
jolt-core/stdlib file as a baked string literal + a cli.ss-style launcher, then
(in a fresh Chez, so the inlined runtime's redefinition of error doesn't strand
early references and runtime eval still sees the runtime's top-level procedures)
compiles it and cc-links it with the Chez petite/scheme boots and the launcher
stub embedded as C arrays. The launcher reads those arrays via FFI on
(jolt-materialize-bundles!) and registers them so build-self-contained can spill
them. joltc itself is cc-linked (clean signature for Homebrew); only the apps it
later builds use the appended-stub path.
build.ss: skip the csv toolchain check on the self-contained path and create the
build dir with a subprocess-free bld-mkdir-p, so a from the
distributed binary shells out to nothing.
release = optimize-level 3 + no inspector info + compressed; debug =
optimize-level 0 + inspector + procedure source + debug-on-exception.
joltc-selfbuild-smoke.sh (make joltcsmoke) builds joltc and, with an empty
environment (no chez/cc/PATH), drives it through the build-app fixture, asserting
the produced binary's output. .gitignore ignores target/.
Adds the pieces a toolchain-free joltc needs to compile apps with no external
Chez or cc:
- host/chez/java/io.ss: register-embedded-bytes!/jolt-embedded-bytes,
read-file-bytes, jolt-spill-embedded!, jolt-append-payload! (frames an app
boot onto the stub as [stub][boot][len:le64]["JOLTBOOT"]), and jolt-chmod-755
via load-shared-object #f (no subprocess).
- host/chez/stub/launcher.c: a native stub that locates its own executable,
reads the trailing frame, and hands the appended boot to the Chez kernel.
- host/chez/loader.ss: resolve-on-roots consults the embedded source store before
disk; ldr-read-source reads baked source. Dev (empty store) is unaffected.
- host/chez/build.ss: build-binary step 4 splits into build-self-contained
(in-process compile-file/make-boot-file with the system error restored, then
append the boot to a copy of the embedded stub) and build-with-cc (the existing
dev path). The self-contained path is taken only when the stub is embedded.
The legacy cc path is unchanged behaviorally; make buildsmoke still passes.
