Auditing the remaining cts baseline for R7 exposed real contract gaps hiding
among the model residue — all fixed to reference behavior:
- stale no-ratio-era stubs: numerator/denominator now work over jolt's exact
rationals (non-ratio is the Ratio cast failure); rational? includes decimals
- casts and pending: peek/pop demand an IPersistentStack (pop nil is nil),
realized? demands an IPending (a plain list/range throws), transient demands
an editable COLLECTION (non-colls throw; the RFC 0003 sorted/list/seq
superset keeps the copy-on-write fallback), empty on a plain record throws
- nil and empties: (nth nil i) is nil, (nth nil i d) is d, a nil index is NPE,
keys/vals of anything empty are nil, (conj nil) is nil
- lookups: contains? on a string is index-only (other keys IAE), get on an
array is lenient (nth still throws), a VECTOR invocation has nth semantics
(([1 2] 5) throws — call position and jolt-invoke both)
- into only transients editable collections; a PersistentQueue/sorted target
folds through conj (RT's IEditableCollection split)
- numbers: number?/num accept BigDecimal, quot/rem throw on an Infinite/NaN
quotient, even?/odd? demand integers
- ordering: keywords compare namespace-first with nil first (Symbol.compareTo)
- misc: run! honors reduced, eval self-evaluates non-form values, intern
demands an existing namespace, counted? excludes strings, seqable? includes
arrays, shuffle rejects maps, sort-by rejects a collection comparator,
when-let demands one binding pair, case*/deftype*/letfn*/reify*/& are
special symbols
Two mis-certified corpus rows fixed (they threw on the JVM too and hid in the
tolerated bucket): a raw \d string escape and duplicate literal map keys.
SPEC.md gains the baseline-traceability section: every one of the 146
remaining suite failures maps to a documented divergence (integer-box,
no-single-float, RFC 0003 transients, seq/chunking model, stm-refs,
parse-uuid strictness, vec-array adoption). cts baseline 5955 -> 6042 pass,
5 errors, 30 namespaces. 9 JVM-certified corpus rows.
The reader now rejects what the JVM reader rejects: a token that starts
like a number but doesn't parse is NumberFormatException (1a, 08, 0x2g,
2r2 — never a symbol); ratio parts are digit runs (1/-1 invalid) with a
zero denominator throwing ArithmeticException; empty ns/name parts are
invalid tokens (:, ::, foo/, /foo) while /, ns//, and :/ stay valid;
duplicate map keys and set elements throw at read; unsupported string
escapes and octal escapes past \377 throw; a stray close delimiter is
'Unmatched delimiter'; \r ends line comments. #inst validates its
calendar fields progressively (leap years included) and #uuid demands
canonical hex. 1-arg symbol splits its ns at the FIRST slash
(Symbol.intern): (symbol "foo/bar/baz") is foo/"bar/baz".
clojure.edn gets its own strict seam (__read-form-edn): auto-resolved
keywords are invalid there, every #_ discarded form validates through the
same :readers/:default pipeline (an unreadable tagged element throws even
when discarded), built-in tags win over :default, M literals construct
BigDecimals, lists satisfy list?, and EOF honors :eof — an opts map
without :eof makes end-of-input an error.
clojure.edn-test.read-string goes 246 pass / 46 fail / 5 errors -> 297/0/0
(fully clean). cts baseline 5904 -> 5955 pass, 23 errors, 56 baselined
namespaces. 9 JVM-certified corpus rows; reader spec section.
The clojure.string case fns and searches now take any Object s through its
toString like the reference's ^CharSequence signatures ((upper-case :kw) is
":KW", (capitalize 1) is "1"); nil throws, and a nil substr in
starts-with?/ends-with? throws. some-fn re-ported with the reference
arities: (some-fn) is an arity error and a no-match result is the last
predicate's own falsy value (false, not nil). ifn? covers multimethods,
promises (which are now invocable — calling one delivers, via a cold-path
invoke-arm registry that costs the hot dispatch nothing), and deftypes
implementing IFn's invoke.
One structural find on the way: defmulti/defmethod deferred inside a fn
body (the deftest pattern) interned/resolved in whatever namespace was
current when they RAN, not the one they were written in — the macros now
bake their expansion ns and the setups honor it.
Also: Boolean/Integer/Double wrapper ctors, primitive TYPE statics
(Integer/TYPE etc.), .reduce on collections (IReduce), and Long/TYPE.
cts baseline 5857 -> 5904 pass, 58 -> 28 errors, 57 baselined namespaces —
the string cluster, some-fn, ifn-qmark, boolean-qmark, and reduce
namespaces are all fully clean. 7 JVM-certified corpus rows; spec entry.
byte/short/int/long/char silently wrapped or passed out-of-range values
through; the JVM range-checks (RT.byteCast family). One checked-cast
helper now carries the ranges: a double range-checks ITSELF before
truncating ((byte 1.1) is 1, (byte 127.000001) throws), NaN casts to 0,
ratios and bigdecs truncate, a non-number is CCE, and the throw carries
the JVM message. float range-checks against Float/MAX_VALUE. The
unchecked-* casts now genuinely wrap and sign-fold ((unchecked-byte 200)
is -56 — the old bit-and lost the sign) with doubles saturating like
Java's conversions; unchecked-long/int are host natives. double/float of
a bigdec convert instead of crashing. The no-single-float residue stays
accepted (SPEC.md).
Also fixes#290: a binary built by the SELF-CONTAINED joltc died with
'variable var-deref is not bound' when a namespace loaded at runtime.
The in-process build compiled flat.ss against a clean copy-environment,
which orphans every top-level define in locations the binary's runtime
eval can't see. It now compiles against the default interaction
environment (defines land in the real symbol cells, same as the legacy
fresh-Chez path) and a generated prologue pre-binds each kernel name the
runtime redefines to its kernel value, so the earliest boot reads match
the legacy path's primitive references. requiring-resolve is implemented
(the issue's dynamic-require pattern), and the release workflow smokes a
runtime require in a built binary.
Cast namespaces byte/short/int/long/char now fully clean; cts baseline
5805 -> 5857 pass, 67 baselined namespaces. 7 JVM-certified corpus rows.
add-watch/remove-watch/set-validator!/get-validator were atom-only; the
atom ctor ignored :meta and :validator; watching a var crashed. Now the
ARef contract is one seam: atoms keep their record slots (hot path
unchanged), every other reference type registers a predicate and stores
watches/validators in identity-keyed side tables, and notifies at its
mutation points. Vars notify on root changes (def on a watched var,
var-set outside a thread binding, alter-var-root — thread-binding sets
don't notify, like the JVM); agents notify per action. The def-var! wrap
costs two weak-table probes per def and does IRef work only on a watched
var.
Ctor options follow ARef: the validator gates the initial value
(IllegalStateException 'Invalid reference state' — also the class for
rejected swap!/reset!), :meta must be a map (else ClassCastException),
nil allowed. meta reads any reference through the identity side-table
(the type-gated fall-through is gone); alter-meta!/reset-meta! work on
non-var references.
Runtime-only (no re-mint). 9 JVM-certified corpus rows; spec entry; cts
baseline 5781 -> 5805 pass, 73 baselined namespaces (the residual error
in the watch namespaces is their STM ref section — refs stay out of
scope).
derive/underive/ancestors/descendants/parents/isa? re-ported from
clojure.core with the argument assertions and throw contracts intact:
derive asserts tag/parent shapes (AssertionError) and throws on redundant
or cyclic derivation; underive/derive on a non-hierarchy value throw at the
parents lookup (the map is called as a function, like the reference);
(descendants h SomeClass) throws UnsupportedOperationException. isa? gains
the reference's supers arm (a relationship derived on a class's super
applies to the class).
The class arms now answer fully through the one class graph: parents of a
class are its direct supers (bases), ancestors are the transitive set
rooted at java.lang.Object for concrete classes (interfaces are marked and
don't root at Object, matching getSuperclass semantics). deftype/defrecord
classes register into the graph at definition — protocol interfaces they
implement appear as supers (JVM-munged ns spelling), records carry the
record interfaces (IRecord/IPersistentMap/... whose closure supplies
Associative/Seqable), bare deftypes carry IType. The type NAME var still
holds the ctor (a jolt-ism); class-key maps it back to the class so
(ancestors TypeName)/(isa? x TypeName) work. canonical-host-tag learned to
NOT canonicalize deftype names through the graph arm (extend-type on a
deftype was registering under the bare segment its values never report).
Five old corpus rows used non-namespaced derive tags that throw on the JVM
too; now namespaced. 8 new JVM-certified corpus rows; spec entries for the
hierarchy family; cts baseline 5730 -> 5781 pass (ancestors/derive/
descendants/parents/underive namespaces fully clean), 74 baselined
namespaces.
Arithmetic and comparisons lowered to raw Chez ops, so an operand outside
Chez's tower (BigDecimal) crashed with a raw condition, and Chez contagion
leaked: (* 1.0 0) gave exact 0 where the JVM gives 0.0, (* ##Inf 0) gave 0
instead of ##NaN, (/ 1 0) raised an untyped error.
One seam now (host/chez/seq.ss): call position emits jolt-n* macros with the
both-Chez-numbers fast path open-coded; value position folds through the same
binary ops. Anything outside the tower falls to per-op slow hooks that
java/bigdec.ss extends, so bigdec arithmetic works in every position (the old
static-only :bigdec typing limitation is gone). JVM rules patched into the
fast path: a double operand wins, an exact zero divisor throws
ArithmeticException while a double zero divisor yields Inf/NaN, quot/rem/mod
cover ratios and doubles, min/max return the original operand with NaN
winning, a nil operand is NPE and a non-number CCE, zero-arg -// throw
ArityException at runtime instead of failing expansion.
Also: with-precision now binds *math-context* and bigdec results round with
real RoundingMode semantics (UNNECESSARY throws; division rounds to precision
instead of throwing); rationalize goes through the shortest decimal print
like BigDecimal.valueOf (the identity stub is gone); ratios coerce to bigdec
like Numbers.toBigDecimal; min/max int-literal operands no longer coerce to
flonum in the numeric pass.
Perf neutral: fib and seq benches unchanged (the fast path is two type checks
the optimizer folds); hinted fl/fx paths untouched. 19 JVM-certified corpus
rows; cts baseline 5614->5730 pass, 192->88 errors, 84->79 baselined
namespaces.
jank-lang/clojure-test-suite (per-core-fn clojure.test suites shared across
Clojure dialects) joins the default gate as vendor/clojure-test-suite, run by
host/chez/cts.sh: one joltc process per test namespace (a hang or crash is
contained by a per-process timeout), through the test/chez/cts-app project and
its cts-run runner, parallel workers.
Gating is exact per namespace against test/chez/cts-known-failures.txt, like
certify's allowlist: a namespace doing worse than the baseline fails, and one
doing better also fails as stale until the baseline is updated in the same
change. JOLT_CTS_WRITE_BASELINE=1 regenerates it; JOLT_CTS_NS runs a subset
verbosely.
Current standing: 243 namespaces, 5302 assertions pass, 340 fail + 236 error
across 88 namespaces pinned in the baseline (dominant clusters: BigDecimal
arithmetic operands, derive/ancestors hierarchy, transients, special-symbol?,
clojure.string case fns, the accepted narrow-int and seq-type-model
divergences). Two consecutive full runs produce identical counts. Wired into
make ci; skips cleanly when the submodule isn't checked out.
Five fixes shaken out by running jank-lang/clojure-test-suite:
- = short-circuits on identity like Util.equiv's k1 == k2, so (= s s) on an
infinite lazy seq answers true instead of walking forever. Numbers keep the
exactness-aware arm ((= ##NaN ##NaN) stays false like the JVM's).
- Calling a non-fn names the operator's CLASS in the ClassCastException, like
the JVM — never the value, whose printed form may be unbounded: ((range))
must throw, not hang rendering an infinite seq.
- realized? on a seq cell answers by its forced flag (the rest of a realized
lazy chain is a cseq under jolt's seq model), and the overlay's unsupported-
type error names the class, not the (possibly infinite) value.
- clojure.test/is dispatches a REGISTERED assert-expr method before its by-name
inline paths, like clojure.test where the built-ins are just pre-registered
methods — so an alias-qualified p/thrown? (the suite's portability helper)
isn't captured by the built-in thrown? path, which read its body as a class.
- clojure.test tracks tests and fixtures per namespace: deftest records its
defining ns, use-fixtures registers under the calling ns (no more cross-ns
clobbering), (run-tests 'ns ...) runs only those namespaces like clojure.test,
and each run-tests call prints/returns its own summary (global counters stay
cumulative for the n-pass/n-fail harness API).
Re-mint (20-coll.clj is seed; prelude only). +2 JVM-certified corpus rows;
the clojure-test fixture pins the alias-qualified assert-expr, per-call
summaries, and ns filtering.
The self-contained build reads the bundled Chez petite/scheme boots from the
joltc binary via foreign-entry on the embedded jolt_* symbols. On Linux dlsym
can't see an executable's symbols unless they're in the dynamic symbol table, so
'build' died with 'foreign-entry: no entry for jolt_petite_boot_len'. -rdynamic
exports them (macOS already resolves them). The new release self-contained-build
smoke caught this on the Linux runner.
`joltc build` inlines the runtime (host/chez/rt.ss and everything it loads, the
seed, compile-eval, loader, ffi, the vendored irregex) into each app binary by
reading those files off disk. That works from a jolt checkout but not from the
installed self-contained binary, which has no source tree:
joltc build -m app.core
=> Exception in call-with-input-file: failed for host/chez/rt.ss: no such file
build-joltc now bakes the exact transitive closure of files the build inlines
into the binary as embedded resources (keyed by the path the `(load "…")` forms
use), and build.ss/dce.ss read runtime source through bld-source-string, which
takes the embedded copy when present and falls back to disk otherwise. So the
same joltc builds apps both from a checkout and standalone.
The release workflow now smoke-tests a self-contained build (compile a tiny app
from an isolated dir, run it) — this is exactly what shipped broken, so it now
gates the release. buildsmoke/shakesmoke/staticnativesmoke unchanged and green.
Build tooling only — no re-mint, no runtime change.
An unknown reader tag produced the reader's internal form
{:jolt/type :jolt/tagged :tag :#foo :form bar}, which tagged-literal? didn't
recognize and which leaked as a raw map when printed:
(tagged-literal? (read-string "#foo bar")) => false ; want true
(pr-str (quote [#foo bar])) => "[{:jolt/type :jolt/tagged ...}]"
Both the data path (rdr-construct-tag) and the compile path (emit-quoted) now
build a real tagged-literal for a tag with no registered reader, like Clojure's
*default-data-reader-fn*, so tagged-literal? / :tag / :form / printing all work.
clojure.edn reads raw forms through a separate __read-form-raw path and applies
:readers/:default itself, so it is unaffected.
Re-mint (backend + reader are seed sources); prelude byte-identical, image only.
make test green (selfhost holds, 0 new/stale), +2 unit rows.
jolt's own throw sites raised untyped Chez conditions with the class name buried
in an English message, so (class e) reported the opaque :object and only a broad
catch worked:
(class (try (Long/parseLong "xyz") (catch Throwable e e))) => :object
; JVM: java.lang.NumberFormatException
Raise typed throwables (jolt-host-throwable) at the Long/Double parse and
StringTokenizer sites so (class e) / .getMessage / a specific catch all reflect
the real class. And fold the exception supertype table (exception-parent) into
the one class graph: exception-isa? now resolves the simple name to its graph key
and asks jch-isa?, so exceptions and every other class share a single hierarchy.
Runtime only, no re-mint. make test green (0 new/stale), +2 corpus rows.
The "does value V declare method M; if so call it" decision was re-derived in a
dozen places with two different lookup helpers — records.ss jrec-cl and
collections.ss rec-coll-method were a byte-for-byte duplicate — and reduce only
honored a reify's own reduce method, not a deftype's:
(reduce + 100 (->Rng 5)) ; Rng deftype implementing IReduceInit
=> "not seqable" ; JVM: 110
Add iface-method / iface-call: one lookup that resolves a method for a deftype OR
a reify, with arity, and is the seam a core fn's interface arm collapses to.
jrec-cl now aliases rec-coll-method (the duplicate is gone). reduce routes its
IReduceInit arm through iface-method, so a deftype's reduce drives the reduction
like a reify's, reduced short-circuit included.
Runtime only, no re-mint. make test green (0 new/stale), +2 corpus rows.
record-method-dispatch was rebound with (set! record-method-dispatch ...) in six
files, each wrapping the previous binding, so precedence was whatever the rt.ss
load order happened to be — the true outermost arm was inst-time's Date arm, not
the one you'd guess. A type-gated wrapper that only whitelists its own methods
then errored on everything else, stealing universal Object methods from the arms
beneath it: (.getClass (java.util.Date.)) threw "No method getClass on Date",
same for File, while (class ...) and (.getClass "s") worked.
Replace the wrapper stack with an ordered list of arms (register-method-arm!,
ascending priority), each returning 'pass to defer. getClass is now one arm at
the top reached by every value, so it can't be shadowed; the three duplicate
getClass checks (dot-forms, host-static, base) collapse into it. Each former
wrapper is an arm at an explicit priority instead of an implicit load-order slot.
A library can register its own arm rather than set!-wrapping the dispatcher.
Runtime only, no re-mint. make test green (0 new/stale divergences), +1 corpus
row for getClass on Date/File.
instance?, extend-protocol dispatch, isa?/supers/ancestors, and the exception
hierarchy each read their own hand-kept table, and those tables had drifted:
(instance? clojure.lang.Associative [1 2]) was true but a protocol extended to
Associative wouldn't dispatch to a vector; keyword/IFn and seq/Seqable had the
same split; (isa? ExceptionInfo RuntimeException) was false and
(supers NumberFormatException) was empty.
Add one FQN -> direct-supers graph (class-hierarchy.ss) and derive the views
from it. value-host-tags builds on the graph closure so a vector reports
Associative/Indexed/ILookup/Counted/Seqable, a keyword reports IFn, a seq
reports Seqable/List/Counted, etc. instance? now tests membership in that same
list, so it can't disagree with dispatch. canonical-host-tag recognizes any
modeled class (was a separate literal set missing Seqable/ILookup/...).
class-direct-supers unions the graph edges and class-supers returns the
transitive closure, so the exception hierarchy answers isa?/supers/ancestors.
The graph is open: jolt.host/register-class-supers! lets a library graft its
own classes on and get every view for free.
Runtime only, no re-mint. make test green (0 new/stale divergences), +3
JVM-certified corpus rows.
joltc failed to start on Windows — "Exception in foreign-procedure: no entry for
pthread_sigmask". concurrency.ss resolves pthread_sigmask/sigemptyset/sigaddset at
load with a top-level (foreign-procedure …), which resolves its symbol eagerly;
those POSIX signal fns don't exist on Windows, so the whole runtime aborted.
Guard the three resolutions (like sched_yield/chmod already are) so a non-POSIX
host yields #f, and make jolt-set-sigint-blocked a no-op when they're unavailable.
The per-thread SIGINT mask is a POSIX-only optimization for the nREPL accept loop;
Windows delivers ^C through the console, and park-until-interrupt still parks on a
condition variable. macOS/Linux resolve the symbols as before — unchanged.
Running the whole rewrite-clj test suite (159 tests) surfaced seven more bugs;
with these it passes 3377/0/0. Each is a general jolt/JVM divergence:
- *out* was pinned to the startup stdout port, so (.write *out* …) escaped a
with-out-str capture (z/print writes via *out*). It now resolves the live
current-output-port, like print/__write, so a redirect is seen.
- nth / assoc past the end of a vector or seq threw a bare Chez error (class
:object). Throw IndexOutOfBoundsException, matching the JVM.
- A number's .toString(radix) ignored the base. Render in the base, lowercase
(rewrite-clj rebuilds 0xff / 0377 / 2r1001 through it).
- A required namespace's own :as aliases leaked into its requirer: the loaded ns
form compiles while (chez-current-ns) is still the requirer, so ce-scan-requires!
registered the loaded ns's aliases under the wrong ns and clobbered a same-named
alias there. Register an (ns NAME …) form's aliases under NAME.
- A quoted collection dropped its metadata; now it keeps USER metadata (drops the
reader's :line/:column/:file), like a Clojure quoted constant.
- enumeration-seq only did (seq e); it now drives a java.util.Enumeration through
hasMoreElements/nextElement, and StringTokenizer implements them.
Regressions: corpus rows (with-out-str/*out*, nth/assoc bounds, toString radix,
quote metadata, enumeration-seq) certified against JVM; a smoke fixture for the
alias leak (a required ns's alias must not leak). tools.reader + rewrite-clj added
to docs/libraries.md. make test green.
irregex builds a POSIX leftmost-longest DFA for a pattern when it can, and jolt
used it for everything. For a pattern with an alternation whose branches have
capturing groups, that DFA leaks a non-participating branch's group: e.g.
#"(?:([0-9])|([0-9])r([0-9]+))" on "2r11" left group 1 = "2" instead of nil, so
tools.reader (rewrite-clj's dep) misread 2r1100 as 2 and 16rFF as 16.
java.util.regex is itself a leftmost-first backtracking engine, so compile a
capturing pattern with irregex's backtracking matcher ('backtrack): its submatch
semantics match the JVM and it clears a losing branch's group. Non-capturing
patterns keep the DFA — with no groups to read, its whole-match result is all a
caller sees, and it avoids backtracking's worst case. The submatch count comes
from a first cheap compile; a capturing pattern recompiles once and caches.
This clears the last rewrite-clj parser-test failure (now 772/0/0). Corpus rows
for the alternation-group case and the radix read. make test green.
Running the rewrite-clj test suite under jolt exposed six bugs, each fixed here:
- `for`/`doseq` `:let` bindings never went through `destructure`, so a
destructuring pattern (`:let [{:keys [y]} x]`) hit `let*` raw and failed to
compile. Emit `let`, like Clojure.
- `with-open` couldn't close a deftype/defrecord that implements a `close` method
(java.io.Closeable / AutoCloseable, e.g. tools.reader's readers) — `__close`
only knew jhost readers and map `:close` fns. Dispatch a record's `close`.
- A deftype/defrecord method param named like a field didn't shadow the field
(the field's let-binding wrapped the params). Params now shadow, as in Clojure.
- A deftype whose simple name collided with a built-in host class clobbered it in
the global ctor table, so `(java.io.PushbackReader. …)` built tools.reader's
same-named deftype. Register deftypes/built-ins by FQN, don't let a deftype
overwrite a built-in's simple name, and qualify a bare `(Name. …)` to the
deftype's FQN only in the ns that defined it.
- `clojure.walk` was lazy over a non-list seq (missing `doall`), so a walk whose
fn has side effects read stale state. Make it eager, like Clojure.
- `Character/isWhitespace` used an ASCII-only check that missed U+2028 and other
Unicode whitespace. Use the JVM's Unicode set (minus the no-break spaces it
excludes).
Regressions: corpus rows (for-let destructure, method-param shadow, walk eager,
isWhitespace), a unit row (with-open closes a record), and smoke checks (the
class-name collision, run in a fresh -e process so the deftype doesn't leak).
One divergence remains unfixed: a submatch from a losing regex alternation branch
leaks when the winning branch has a quantified group (a bug in the vendored
irregex engine, not jolt) — tracked separately.
A registered #tag data reader whose fn returns a FORM (borkdude/html's #html
expands to (->Html (str …))) was rewritten to a runtime call (reader-fn 'inner),
so the returned code became a runtime list value instead of being compiled —
(str #html [:div]) rendered the code, not "<div>". Clojure applies a data reader
at read time and substitutes its result as code.
loader.ss now applies the reader at load time: a code form (a list) is spliced in
to be compiled, a value (time-literals #time/date -> a Date) keeps the runtime
call, which also keeps a non-serializable constant out of an AOT build. The build
emit path never applied data readers at all (a #tag literal failed a `jolt build`
with "unsupported form"); emit-image.ss gets an ei-emit-form-hook the build sets
to the same rewrite, left as a no-op elsewhere so the seed mint (which doesn't
load loader.ss) is unaffected and the self-host byte-fixpoint holds.
Also make clojure.test report the actual values of a failing (is (= a b)) — it
printed only the form. Restricted to the common pure predicates so a macro head
still takes the plain path.
Fixture test/chez/datareader-app + a smoke check (interpreted) and a build-smoke
check (AOT). make test green, no corpus change.
A :jolt/native spec can now carry a :static archive; `jolt build` links it
into the executable, so the app calls the C code with no shared object on the
target. --dynamic (or :jolt/build {:dynamic-natives true}) keeps the old
runtime load-shared-object behavior; a spec with no :static is unchanged.
The cc link force-loads the archive (-force_load on macOS, --whole-archive on
Linux) and exports the executable's symbols (-rdynamic on Linux) so the baked-in
symbols resolve via (load-shared-object #f) + foreign-procedure at startup. Build
step 1 evaluates the app's foreign-procedure forms in-process, so a static
archive is preloaded there as a throwaway shared object to resolve them.
The distributed self-contained joltc has no external cc/Chez but must build these
apps, so it now bundles the Chez kernel (libkernel.a + scheme.h) and the launcher
source and re-links a custom stub with the archives baked in — needing only a
system cc, no Chez. run/repl skip static-only specs (nothing to load); keep a
:darwin/:linux candidate to use such a lib interpreted.
Adds static-native-smoke (cc path) to ci and a static phase to the joltc
self-build smoke (distributed path).
repl-form-complete? entered the :regex state on '#' but only consumed the
'#', so the opening '"' was then read by the :regex handler as the CLOSING
quote. The regex body got scanned in :code state, and any delimiter or quote
inside it (a group like #"(a)", a char class #"[0-9]+") threw off the
paren/string count — so a one-line regex form was judged incomplete and the
REPL hung waiting for continuation lines. Consume the '#"' together.
Adds a self-checking predicate test (test/chez/repl-reader-test.clj, run via
joltc so jolt.main resolves) and an end-to-end regex REPL case in smoke.sh.
The REPL evaluated one line at a time, so a form split across lines
(e.g. `(+` then `1 2)`) raised instead of waiting. The read loop now
accumulates lines until delimiters are balanced — skipping string,
char, regex and comment context — printing a `... ` continuation prompt
for each extra line.
Reader/runtime errors rendered as Chez's "attempt to apply
non-procedure #[chez-pmap...]" instead of their real message. Two causes:
jolt-throw raised the thrown value raw. When a throw crossed the host
`eval` boundary, Chez re-wrapped the non-condition into a compound
condition whose message extraction applies the value, losing the message
and crashing on ex-info's empty-map :data. jolt-throw now raises a
&jolt-throw condition wrapping the value; catch (lowered to `guard`),
jolt-report-uncaught and jolt-render-throwable unwrap it back via
jolt-unwrap-throw, so ex-data/ex-message and the backtrace tag survive.
Every reader/post-prelude EOF-throw site used `(empty-pmap)` (with
parens), applying the empty-map value as a procedure and crashing during
ex-info construction before jolt-throw ran. Fixed to `empty-pmap`.
Re-minted the seed; smoke 23/23, unit 574/574.
^C to a running `joltc --nrepl-server` aborted with "thread does not
own mutex" because the accept-loop thread absorbed SIGINT in its foreign
accept() call, where Chez can't run the keyboard-interrupt handler, and
run-main-pump's tight condition-wait loop wasn't interruptible anyway.
Block SIGINT in the primordial thread before starting the server so the
accept loop inherits a blocked mask, park in a single interruptible
condition-wait via the new park-until-interrupt, and run registered
shutdown hooks (newest-first, each isolated) from the keyboard-interrupt
handler before (exit 0). The stop fn now drops .nrepl-port via the new
jolt.host/delete-file seam — clojure.java.io/delete-file doesn't exist
in Jolt and silently no-ops, so .nrepl-port was never removed.
^D (EOF) exits cleanly in canonical mode but some terminals and editors
don't deliver it, leaving the user stuck. Accepting :repl/quit or :exit
as the first form of a line gives a reliable keyword exit that works
everywhere. The check parses the line with read-string rather than
checking the evaluated value, so a nested value that happens to print
as the keyword can't trigger an exit.
list?, ratio?, and rational? are the predicate-web members that are
genuinely safe to migrate: not extended at runtime, not on the compiler
emit/inference path, not reached by the kernel tier. They now live in the
overlay (clojure/core/20-coll.clj) built on the jolt.host tower/rep tests,
lowering to the same code the native shims did. Removed their native
definitions (predicates.ss) and, for ratio?/rational?, the now-redundant
post-prelude re-assertions. Also dropped the dead all-flonum overlay
ratio?/rational?/decimal? stubs.
The rest of the web stays native and is documented as such: map?/set?/
seq?/coll? are extended with sorted/record/lazy arms, decimal? is extended
by the optional bigdec module, integer?/float? are on the emit/inference
path, vector? is reached by the kernel-tier peek. jolt.host exposure is
therefore narrowed to just the tests these three consume (exact?,
rational-type?, cseq?, cseq-list?, empty-list?).
Numeric probe is byte-identical to pre-migration; list? correct across
list/vector/lazy/empty/cons/rest cases. Selfhost fixpoint holds, values/
unit/smoke/corpus green, bench flat within noise.
The clojure.core type predicates bottom out at host tests that overlay
Clojure can't reach. Expose them under jolt.host so the predicate web can
be built as pure compositions that lower to exactly these calls:
numeric tower: exact? flonum? integer-type? rational-type?
collection reps: pvec? pmap? pset? cseq? empty-list? cseq-list? lazyseq?
exact? is wrapped to be total (Chez's raw exact? errors on a non-number;
the others return #f for a non-match). lazyseq? is exposed in
lazy-bridge.ss because jolt-lazyseq? is defined there, after predicates.ss.
map?/set?/seq? are deliberately not reduced to a single rep test: they are
extended at runtime with sorted-collection/record/lazy arms, so only the
rep predicates are exposed, not those unions. Additive only (new bindings,
nothing references them yet); bench unchanged within noise.
Attempting to migrate the reader-conditional constructor to the overlay
revealed that an overlay defn returning a :jolt/type-tagged map literal
silently fails to bind during the seed mint: the guard around each
prelude form swallows the load-time error, leaving the var unbound. This
is the same reason every other tagged-value constructor (atom,
volatile!, tagged-literal) is native, so reader-conditional is
reclassified STAY-PRIMITIVE rather than a safe migration.
set was a native shim (apply jolt-hash-set (seq->list coll)). It is a
pure composition, so the Clojure version (apply hash-set (seq coll))
lowers to the same code. The compiler uses set, but only off the emit
path (the backend's bare-native-names def and type inference), so it can
live in the kernel tier: compiling that tier never calls set, and by the
time those callers run the tier is already bound.
This is distinct from boolean, which the backend calls for every :if
node on the emit path. Moving boolean even to the kernel tier deadlocks
(compiling the tier that defines boolean needs boolean), so boolean stays
native. Added a comment in predicates.ss recording that.
Re-mint converges in 3 passes and the benchmark suite is unchanged
within noise (collections 43.3 vs 43.1, binary-trees 367 vs 367, the
rest flat).
The overlay defines transduce in clojure/core/22-coll.clj as a pure
composition (xf (reduce xf init coll)), and it shadows the native
jolt-transduce by load order. The compiled overlay version is already
what gets baked into the seed, so the native binding in
natives-transduce.ss was dead weight.
transduce is not used by the self-hosted compiler and no overlay tier
before 22-coll references it, so removing the native binding is safe.
Re-minting produces a byte-identical seed, which proves the runtime is
unchanged. sequence stays native (its transformer iterator drives the
reduced box and lazy realization directly).
range, map, and filter were fully element-by-element lazy, so
(map f (range 1 50)) realized one element per first/nth where JVM
Clojure realizes a whole 32-element chunk. range is a chunked
LongRange on the JVM and map/filter are chunk-preserving, so the
observable side-effect timing differed.
Following clojure.lang.LongRange, ChunkedCons, ChunkBuffer and
core.clj, this adds a crest field to the cseq record and a
cseq-chunked constructor modeling ChunkedCons (a standalone chunk
pvec, an offset, and the after-chunk seq). The chunk accessors move
to seq.ss next to the representation they read. map/filter/remove
take a chunked branch when the source is chunked, realizing the whole
chunk and chunk-cons'ing it onto a lazy rest, so their output is
itself chunked and chained transforms each batch by 32. Bounded range
is now an eager chunked seq, and the reduce fast path flows through a
ChunkedCons rest. The chunk-buffer/chunk/chunk-cons builder API in
natives-array.ss now produces a real ChunkedCons.
Single-arg (range), multi-coll map, and plain lazy seqs stay
element-by-element, like the JVM.
Adds a lazy / chunking suite to the corpus that observes realization
timing via an atom counter: first over a chunked map realizes 32,
crossing a chunk boundary realizes 49, chained maps batch [32 32],
filter applies the predicate to the whole first block, and a plain
lazy seq still realizes one element at a time. Two cases that
documented the old over-laziness now assert the JVM value of 32 and
were dropped from the allowlist. certify against JVM Clojure 1.12.3
reports 0 new and 0 stale divergences.
host/chez/build-joltc.ss builds joltc into target/<profile>/joltc: it emits a
flat source of the full runtime + compiler image + inlined build.ss + every
jolt-core/stdlib file as a baked string literal + a cli.ss-style launcher, then
(in a fresh Chez, so the inlined runtime's redefinition of error doesn't strand
early references and runtime eval still sees the runtime's top-level procedures)
compiles it and cc-links it with the Chez petite/scheme boots and the launcher
stub embedded as C arrays. The launcher reads those arrays via FFI on
(jolt-materialize-bundles!) and registers them so build-self-contained can spill
them. joltc itself is cc-linked (clean signature for Homebrew); only the apps it
later builds use the appended-stub path.
build.ss: skip the csv toolchain check on the self-contained path and create the
build dir with a subprocess-free bld-mkdir-p, so a from the
distributed binary shells out to nothing.
release = optimize-level 3 + no inspector info + compressed; debug =
optimize-level 0 + inspector + procedure source + debug-on-exception.
joltc-selfbuild-smoke.sh (make joltcsmoke) builds joltc and, with an empty
environment (no chez/cc/PATH), drives it through the build-app fixture, asserting
the produced binary's output. .gitignore ignores target/.
Adds the pieces a toolchain-free joltc needs to compile apps with no external
Chez or cc:
- host/chez/java/io.ss: register-embedded-bytes!/jolt-embedded-bytes,
read-file-bytes, jolt-spill-embedded!, jolt-append-payload! (frames an app
boot onto the stub as [stub][boot][len:le64]["JOLTBOOT"]), and jolt-chmod-755
via load-shared-object #f (no subprocess).
- host/chez/stub/launcher.c: a native stub that locates its own executable,
reads the trailing frame, and hands the appended boot to the Chez kernel.
- host/chez/loader.ss: resolve-on-roots consults the embedded source store before
disk; ldr-read-source reads baked source. Dev (empty store) is unaffected.
- host/chez/build.ss: build-binary step 4 splits into build-self-contained
(in-process compile-file/make-boot-file with the system error restored, then
append the boot to a copy of the embedded stub) and build-with-cc (the existing
dev path). The self-contained path is taken only when the stub is embedded.
The legacy cc path is unchanged behaviorally; make buildsmoke still passes.
PR #269 added the main-thread executor (call-on-main-thread, run-main-pump,
stop-main-pump) so the nREPL accept loop can run on a worker thread while the
primordial thread owns the GUI main loop. Two problems made stop-main-pump
unusable as a graceful-shutdown or external API.
run-main-pump set jolt-main-pump-active to #t on entry but never cleared it on
exit, so after the pump returned the flag stayed #t. call-on-main-thread also
read that flag outside the queue mutex, so even with a reset there was a window
where a job could be enqueued just as the pump left, then block forever on a
pump that was gone.
Both are now decided under jolt-main-queue-mu. The pump clears active in the
same critical section where it sees the stop flag and an empty queue, and
call-on-main-thread reads active and enqueues atomically under that lock. A
caller that loses the race sees the pump inactive and runs the thunk inline,
the same fallback used when no pump is running, rather than blocking. A
dynamic-wind around the loop also clears active on an abnormal exit so a later
run-main-pump starts clean.
Match babashka's spelling: the nREPL server now starts with
`bin/joltc --nrepl-server [port]` instead of `bin/joltc nrepl`. Port
parsing and JOLT_NREPL_PORT are unchanged.
Also wire up --help/-h to print usage (previously only the no-arg
invocation did), and fix the usage listing to show the real flag.
Smoke now asserts --help mentions --nrepl-server. Docs updated to match.
Profiling jolt-i5if showed <=60-bit arithmetic is already native-fast; the real
general overhead in the run/-e/-m path is var resolution. Every var reference
compiled to (var-deref ns name), which builds + hashes a fresh "ns/name" string
and does a hashtable lookup per access (~45ns). The var cell is interned and
def-var! mutates it in place, so caching the resolved cell is sound under
redefinition.
Generalize the devirt per-site cache-cell mechanism to var value references: a
ref inside a fn resolves its cell once into the def's closure, then reads it via
var-cell-deref (a field read after the first). var-cell-deref is the cell-based
var-deref — binding-aware (dynamic vars + *ns* still resolve) and lenient on an
unbound root (a forward-declared var doesn't throw, unlike jolt-var-get).
Gated by a runtime flag: ON for runtime-compiled code (compile-eval.ss), OFF for
the seed mint and AOT build (emit-image.ss) so the seed stays a byte-fixpoint --
prelude.ss is unchanged, only image.ss picks up the new backend. ~5x on a
var-ref-heavy loop (1058ms->205ms); ~1.2x on test.check (its generators are more
deftype/dispatch-bound than var-deref-bound). No C/FFI.
Corpus rows pin redefinition / dynamic binding / forward ref through a cached
ref. make test + shakesmoke green, selfhost holds, SCI 211/218, certify 0-new.
Profiling the test.check distribution/large-sample slowness (jolt-i5if): the
hot path is the SplitMix PRNG, dominated by 64-bit mix arithmetic, and the
bitwise ops (bit-and/or/xor/not, shifts) were NOT in the backend native-ops
table — so (bit-xor a b) compiled to a var-deref through the variadic overlay
(__bit-xor) instead of a direct call, the way +/-/* already emit.
Map bit-and/or/xor/not to the Chez bitwise-and/ior/xor/not primitives (inlined
to native code; a non-integer operand now errors like the JVM instead of being
silently truncated) and the shifts to a direct helper call. bit-and-not stays on
its overlay — its only Scheme impl is 2-arg, so a value-position arity-3 use
would mis-emit.
mix-64 arithmetic 2.7x faster, raw split+rand-long 2.4x, gen/vector ~1.4x. The
remaining gap is the bignum-vs-native-long floor (~20x, substrate) plus the
generator machinery (deftype/fn dispatch, separate). Corpus rows added for value
position, bit-not, apply, and a full-64-bit unsigned shift.
jolt's `is` was a fixed macro with no assert-expr multimethod, and the runner
bypassed the report multimethod, so libraries couldn't register custom
assertions or custom report types (e.g. test.check's ::trial/::shrunk).
Add assert-expr (2-arg [msg form], dispatch on the form's first symbol /
:default / :always-fail), do-report routing through report, and report
:pass/:fail/:error methods that feed the counters. `is` dispatches to an
explicitly-registered assert-expr method before its inline path, so thrown?/
thrown-with-msg?/= and every built-in form stay byte-identical.
Runtime stdlib only, no re-mint. test/chez/clojure-test.clj self-checks the
extension points + full is/are/testing/thrown?/use-fixtures surface; smoke gate
runs it.
A pvec is a 32-way trie, but na-chunk-first built each block by calling
pvec-v on the full backing vector — materializing all n elements to a
flat Scheme vector — then copying the 32-wide window out of it. That made
chunk-first O(n), so walking a vector chunk-by-chunk (Clojure's real
chunked map/filter fast path) was O(n^2): a ported chunked map over 500K
elements took 39s, superlinear to ~700s at 2M.
na-chunk-size equals pv-width and blocks are 32-aligned, so a block is
exactly one trie leaf — pv-chunk-for hands it back in O(log n). Copy that
leaf directly; fall back to per-index reads for the rare window that
crosses a leaf boundary. Chunked map is now linear, ~133x faster at 500K
(293ms) and within ~2.3x of the native seq loop, which makes a
clojure-in-clojure seq tier viable.
Corpus rows pin chunk-first window contents + chunk-rest boundaries
against JVM; fixed a stale 'always false' chunked-seq? label.
map/filter/remove/take/drop/concat/take-while/drop-while/mapcat/partition
built an eager-headed cseq: the first element (and the fn application) ran
at construction, so a side-effecting (map f coll) fired f immediately and
(class (map …)) was PersistentList instead of LazySeq. This diverged from
Clojure, which wraps the whole body in lazy-seq. It went unnoticed because
the conformance gate certifies values, not realization — eager and lazy
heads produce identical values — and unit.edn even baked PersistentList in
as expected. test.check's for-all-takes-multiple-expressions (which counts
side effects in a for-all body) exposed it.
Wrap each native producer's result in a lazy-seq node so the body, incl.
the first element, defers until forced — the forced cseq still has eager
heads, so reduce/count/dorun/etc. force on walk and there's no per-element
cost. dedupe's (seq coll) is moved inside its lazy-seq. A jolt LazySeq is
now recognized by coll?/empty, the analyzer's form predicates (a macro can
build its expansion with map), value-host-tags + instance? (LazySeq/ISeq/
Sequential), and reports clojure.lang.LazySeq.
Kept the native Scheme implementations rather than porting Clojure's: a
straight lazy-seq+cons port is 3x slower and Clojure's chunked fast path is
288x slower because jolt's chunk machinery is unoptimized (filed jolt-j9dz);
the wrapped natives are Clojure-lazy at native speed.
+12 corpus rows (laziness at construction, LazySeq type, both JVM-certified).
make test + shakesmoke green, selfhost holds, 0 new divergences.
Chez fixnums are 61-bit, so the bignum bitwise-and mask allocates for any
value past 2^60 — and unchecked-* ran it on every result, even small
in-range ones. An exact integer already in [-2^63, 2^63) is its own wrap,
so return it directly; only an out-of-range result (a multiply overflowing
into 128 bits) needs the mask. ~30% on in-range unchecked-add loops,
neutral on full-64-bit multiply.
Note: the 64-bit arithmetic floor on Chez stays ~31ns/multiply (bignum, no
native 64-bit int); the test.check distribution hangs are dominated by
generator/dispatch overhead, not arithmetic — this is a general win for
long-heavy code, not a fix for those.
jolt-unc{add,sub,mul,inc,dec,neg}2 wrapped every non-flonum result to a
signed 64-bit integer, so (unchecked-add 2/3 2/3) truncated to 1 instead
of 4/3. Under *unchecked-math* the analyzer rewrites +/-/* to unchecked-*,
so any ratio arithmetic in such a file silently floored. Clojure's
unchecked-add falls back to regular arithmetic for non-primitives; only
long math wraps. Wrap iff both operands are exact integers.
Shaken out by test.check's gen/ratio monoid property (the + and 0 monoid
held for small-integers but failed for ratios).
Close clojure.spec.alpha's remaining gaps — its conform/explain/describe/multi-spec
suite (clojure.test-clojure.spec, multi-spec) now passes fully.
- (get reify k) / (:k reify) routes to a reify's clojure.lang.ILookup valAt. spec
reifies fspec/regex specs as ILookup and reads (:args spec) off them, so before
this instrument never saw the args spec.
- A failed numeric comparison reports the JVM class: a nil operand is
NullPointerException, a non-number is ClassCastException (was an opaque :object
condition). conform-explain checks the thrown class.
- A quoted / macro-form #inst / #uuid literal constructs its Date/UUID value, like
the JVM reader (which builds it at read time). emit-quoted was emitting the raw
tagged form, so #inst "1939" and #inst "1939-01-01T00:00:00.000-00:00" weren't =.
- An anonymous fn reports class clojure.lang.AFunction$fn (the $fn marker), so
spec's fn-sym returns ::s/unknown for it, matching the JVM's ns$fn__N.
- A fn with & {:as m} kwargs accepts a trailing map (Clojure 1.11): (f :a 1 {:b 2})
and (f {:a 1}) both bind m, by merging an odd trailing map over the pairs.
- A thread responds to .getStackTrace (empty — jolt does TCO).
clojure.test-clojure.instr does not fully pass: its ::caller assertions need the
calling fn's stack frame, which TCO erases (an inherent host divergence, like the
JVM keeping tail frames).
make test green (+4 corpus rows, 0 new divergences), shakesmoke byte-identical.
Re-mint (backend emit-quoted + the destructure macro).
jolt's reader had no case for #: , so #:event{:type :search} died as an unknown
tagged literal. Now #:ns{...} qualifies each bare keyword/symbol key with ns
(:_/x stays unqualified, an already-qualified key is left alone); #::{...} uses
the current ns and #::alias{...} resolves the alias — matching Clojure.
clojure.spec.alpha's multi-spec test (which builds #:event{...} event maps) now
passes.
make test green (+1 corpus row, 0 new divergences), shakesmoke byte-identical.
One re-mint (the reader is a seed source).
