A jolt library can now bind its own native dependencies and expose a Clojure API
over them — no jolt built-in required. This is the foundation for moving the
http-client / db / adapter functionality out of the host and into real libraries.
- jolt.ffi/foreign-fn (sugar: defcfn) is a compiler special form: a compile-time
-typed C signature lowers to a real Chez foreign-procedure (analyzer :ffi-fn ->
backend foreign-procedure), so calls are typed and marshaled, not eval'd.
- host/chez/ffi.ss provides the rest under jolt.ffi: load-library, alloc/free,
read/write/sizeof, ptr<->string, null/null?. Loaded after the loader snapshot
so a library's (require '[jolt.ffi]) still loads the macro side.
- Types: int/uint/long/ulong/int64/uint64/size_t/ssize_t/iptr/uptr/double/float/
pointer/string/void/uint8/char.
Validated end to end: a pure-Clojure file binds libc (getpid/strlen/abs) and
libsqlite3 (open/prepare/step/column/finalize over out-param pointers) and runs a
query. Gate test test/chez/ffi-binding-test.ss (make ffi); selfhost holds.
Bring the docs in line with the actual implementation now that Chez is the sole
substrate.
Deleted the migration/spike/handoff artifacts that only documented the Janet
era or the port effort: the port plan, phase-0 and foundational-runtime spike
writeups (+ the stray root-level copy), the self-hosting design notes, the
architecture-refactor plan, and spike/chez/RESULTS.md.
Rewrote the current reference docs against the Chez facts: building-and-deps and
tools-deps (no jpm/build step — bin/joltc off the checked-in seed, deps via
jolt.deps into ~/.jolt/gitlibs), libraries (SQLite is built-in jdbc.core over
libsqlite3, not a Janet driver), the conformance/spec test-flow docs (the Chez
corpus runner + certify, no .janet harnesses), and the transient / type-hint /
seed-overlay design notes (Chez representations: mutable transients, flat
copy-on-write vectors, HAMT maps, the seed/overlay twin). Fixed the README
collections line (vectors aren't 32-way tries) and added the ffi/transient gate
targets. rfc 0001's numerics open-question is resolved (the Scheme tower).
Renamed the built-in HTTP adapter to jolt.http.server only (dropped the
ring-janet.adapter alias — a Janet-era name).
Shaking the ring-app example's real library stack out against jolt surfaced a
batch of divergences from JVM Clojure, the biggest being evaluation order.
backend_scheme: call and recur arguments were emitted as bare Scheme operands,
so Chez's unspecified (right-to-left) order won out. Clojure evaluates left to
right, which selmer's reader loop relies on: (recur (add-node ... rdr) (read-char
rdr)) consumed a char early and dropped the first chars of every {{tag}}. Bind
operands to fresh temps in a let* (only when two or more can have side effects,
so hot calls over locals/consts stay un-wrapped). emit-ordered already did this
for collection literals; generalize it.
host-contract: syntax-quote now resolves the alias part of a qualified symbol
(impl/foo -> clojure.tools.logging.impl/foo) instead of leaving it bare, which
limped along via short-name matching until two loaded namespaces (reitit.impl,
clojure.tools.logging.impl) shared the short name and it broke.
collections: key-hash masks with bitwise-and, not fxand — jolt-hash is set!-
decorated per type (records return their own hash) and Chez's equal-hash can be a
bignum, so a key's hash isn't always a fixnum.
seq: even?/odd? handle bignums (JVM accepts any integer; the fxand crashed).
records: Keyword/Symbol .sym/.getName/.toString (honeysql's :clj branch reads
(.sym k)); Throwable .getMessage/.toString over a Chez condition.
host-static: __register-class-ctor!/__register-class-statics! so a host shim
(reitit.trie-jolt) can mirror a Java class.
natives-str: String.intern returns the string.
sqlite: jdbc.core fetch/fetch-one kebab-case column keys (the jolt-lang/db
convention; created_at -> :created-at).
io: a relative io/file path resolves against JOLT_PWD (the user's cwd), not the
repo root the launcher cd'd to — matches JVM cwd semantics, so config.edn loads.
cli: render an uncaught jolt throw (ex-info message + ex-data, or a condition)
instead of Chez's opaque "non-condition value" dump.
Reader / loader:
- #?@ splicing reader conditionals now actually splice the matched collection's
items into the enclosing sequence; the splice flag was read but ignored, so a
binding vector like [a #?@(:clj [b (.foo b)])] lost its alignment.
- the file loader reads by position and skips a top-level form that reads as
nothing (a :cljs-only #?, a #_ discard, a trailing comment) instead of
treating it as EOF — which silently dropped the rest of a large .cljc file.
- jolt's reader feature set now includes :clj (was {:jolt :default}). jolt is a
Clojure/JVM-compatible host that emulates clojure.lang.* and java.* interop,
so it reads the :clj branch of a .cljc library, not :cljs. This also lets four
more reader-conditional corpus cases pass (floor 2726 -> 2730).
Backend:
- munge-name escapes ' (prime) -> _PRIME_; a Clojure symbol like f' otherwise
emitted a bare ' into Scheme, which is the quote reader macro and unbalanced
the output.
Host shims:
- clojure.java.io/writer (pass through a StringWriter, file-back a path) and a
readLine on the string reader, so line-seq over (io/reader …) works (markdown).
A better "unsupported destructuring pattern: <pat>" error message.
deftype fields tagged ^:unsynchronized-mutable / ^:volatile-mutable can now be
reassigned in place from a method, as on the JVM. A jrec stores fields as cons
cells, so a new jolt-set-field! mutates the pair with set-cdr!. The deftype macro
rewrites (set! mutable-field v) in a method body to (set! (.-field inst) v), and
the analyzer compiles a (set! (.-field obj) v) target to jolt-set-field! — so
both the rewritten symbol form and an explicit interop (set! (.-root this) v) go
through one path. Field reads remain a snapshot at method entry, which is correct
for the universal read-then-set pattern (a repeated set! of the same field in one
call would read the entry value).
Closes the set!-of-local SCI failures: SCI load 202 -> 205/218.
Found in a read/eval review: a local named like a special form wrongly took
over operator position. (let [if (fn ...)] (if true 1 2)) returned the fn, but
per spec section 3 (and the reference) special-form heads are not shadowable;
only macros are. Two fixes: drop the (not shadowed) guard on the special-form
branch of analyze-list (so an (if ...) head is always the special), and prefix
a local whose name is a Scheme keyword when emitting (so a value local legally
named if does not shadow the (if ...) the back end emits). Value-position
locals named if/or/case still work.
The analyzer checked special forms before expanding macros, the reverse of the
canonical read -> macroexpand -> analyze order (Clojure/CLJS analyze-seq). Move
macroexpansion to the front of analyze-list. Knock-on fixes:
- letfn was both a (broken) macro expanding to let* AND a primitive special
(analyze-letfn, proper letrec*). Macroexpand-first surfaced the macro, breaking
mutual recursion; remove the macro, keep letfn a primitive.
- defmacro is now compiled by the analyzer (a :set-var-style :defmacro node that
defs the expander fn via the fn macro — so destructuring arglists desugar — and
marks the var a macro), so a non-top-level (when … (defmacro …)) works. The
runtime spine's separate top-level defmacro interception is removed: one path.
SCI load 162 -> 202/218.
The analyzer punted set! as uncompilable. Add it as a special form: (set! sym
val) on a var emits jolt-var-set, which updates the innermost thread binding (or
the root when unbound), returning val. A local target (deftype mutable field) or
an interop (.-field) target stays uncompilable for now. Also defines
*warn-on-reflection* (false) so set! on it resolves. SCI load 186 -> 196/218.
Chez rejects duplicate lambda formals, so any (fn [_ _] ...) failed to compile
— including every macro expander, whose &form/&env slots both expand to _. The
analyzer now renames each earlier duplicate param to a fresh name (Clojure binds
the last occurrence, so the earlier ones are shadowed and unreferenceable). SCI
load 162 -> 186/218.
bigdec / 1.5M / 0.0M silently produced doubles. Add a jbigdec value type
{unscaled, scale} over Chez exact integers (host/chez/bigdec.ss): value =
unscaled * 10^-scale. An M-suffix literal reads to a :bigdec form that the back
end lowers to jolt-bigdec-from-string (same IR-leaf path as #inst/#uuid); bigdec
coerces a number/string. Equality is by value (1.0M = 1.00M true, 3M = 3 false),
str drops the M and pr keeps it, class is java.math.BigDecimal, decimal? is true.
Arithmetic contagion isn't modelled (out of scope). The old corpus cases passed
spuriously as doubles; they now exercise a genuine BigDecimal.
(def ^String tv ...) left (:tag (meta (var tv))) as the unresolved "String";
the JVM compiler resolves the hint to java.lang.String at def time. Add a
resolve-class-hint host seam (built from the existing class-token table) and
resolve a def's :tag through it in the analyzer. The reader path
(read-string "^String x") stays unresolved, matching the JVM (only the
compiler resolves). Closes ^Type-tag-on-var.
(map? *in*) was true because *in* was a plain map of read-line-fn/read-fn
closures; the JVM *in* is a java.io.Reader so map? is false. A defrecord
doesn't help (records are maps). Make the reader a reify over a new IReader
protocol — a non-map value — and route read/read-line/read+string/line-seq
through its -read-line/-read-form/-read+string methods instead of keyword
access. with-in-str's __string-reader and the stdin *in* both reify it.
Closes *in*-bound + *in*-is-bound.
A macro like (defmacro cur-ns [] `(str ~*ns*)) splices the live *ns* value
into its expansion, leaving an opaque jns object as a list element. The
analyzer had no way to carry a runtime value and threw uncompilable — the last
remaining corpus crash. Recognize a jns via the host contract (form-ns-value?)
and emit a :the-ns leaf that reconstructs it by name (intern-ns!) at the call
site, the same IR-leaf pattern as regex/inst/uuid. Closes unquote-*ns*-in-
template; corpus crash count -> 0.
A namespace fast path rather than a general constant pool: it's the only
embedded-value case in the corpus and the common real-world one (libs splice
~*ns*). A general pool can come later if other value types appear.
deftype/defrecord inline protocol methods went through extend-type ->
register-method, so a record implementing a protocol inline showed up in
(extenders P) — the JVM only lists extend/extend-type/extend-protocol
registrations there (inline impls compile into the class). Add
register-inline-method: it registers for dispatch under the record tag but
skips the extender mark. The mark lives inside type-registry so the per-case
corpus prune restores it. Closes corpus lists-extended-type + seq-of-tags.
definterface now expands to (do (def name {}) 'name) so (var? (definterface ...))
is false, matching the JVM where it yields the interface Class. ns-imports returns
the 96 auto-imported java.lang classes (short symbol -> canonical name) so
(count (ns-imports 'user)) is 96. Re-minted for the macro change. Corpus 2718->2720.
Allowlist review found three addressable divergences:
- unchecked-char returned a number; the JVM returns a char.
- the readable printer (pr-str, coll elements, the -e/REPL printer) rendered
infinities as Infinity/inf; Clojure's readable form is ##Inf/##-Inf/##NaN
(str/print still gives Infinity). So (pr-str ##Inf) => ##Inf, (str [##Inf]) =>
[##Inf], (str ##Inf) => Infinity.
Corpus 2695->2698; allowlist 43->40 (drop the 3 now-passing entries). Re-minted.
The . special form rejected a non-symbol member; a keyword member now lowers to
an invoke of the keyword on the target ((. {:value 41} :value) => 41, as on the
JVM). Added a form-keyword? contract seam. Corpus 2692->2693. Re-minted.
(.. x m ...) failed: the analyzer classified the .. head as a .method interop
call (method-head? matched any "."-prefixed name) and form-special?
(hc-interop-head?) also flagged it, so it never reached the macro check. Exclude
".." from both (the char after "." being "." means the threading macro, not
.method). Corpus 2690->2691. Re-minted.
sorted-map seq/first/entries built plain [k v] vectors, so map-entry? was false
and key/val threw. Build them via a new jolt.host/map-entry seam (entry-flagged
pvec), matching a regular map's entries. Re-minted.
(into [] (dedupe) coll) / (sequence (dedupe) coll) threw an arity error — dedupe
was [coll]-only. Add the 0-arg stateful transducer (tracks [seen? prev] in a
volatile, no sentinel). Re-minted.
- == 1-arg returns true for any value (Clojure short-circuits before the number
check), not 'requires numbers'.
- current-time-ms wired to now-millis so the time macro works.
- subvec truncates float/ratio indices via long (Scheme quotient rejects flonums).
- defonce checks bound? not var-get — in a top-level do the name is already an
unbound interned cell, which var-get throws on.
- drop the line-seq corpus row (used janet/spit, N/A); allowlist char-array
(needs Class/forName "[C").
Corpus 2678->2683, floor raised. Re-minted. Full gate green; CI green.
jolt-cf1q.7
Rephrase comments that pointed at deleted Janet files (emit.janet, the seed
sources, 'the Janet back end punts ...') to present-tense descriptions of the
Chez behavior. Comment/docstring-only; the self-host fixpoint is unchanged
(comments don't affect the compiled seed).
Delete five files that were Janet-host shims with no Chez path: clojure.java.io
(provided natively by host/chez/io.ss), and jolt.{nrepl,png,interop,shell}
(the janet.* bridge, os/shell, janet.net — none exist on Chez).
jolt-cf1q.6
jolt was all-flonum (one :number type, inherited from Janet whose only number
type is a double). The Chez runtime has a full numeric tower, so the zero-Janet
path now carries it = JVM Clojure semantics:
(/ 1 2) => 1/2 (exact Ratio, was 0.5)
(integer? 3) => true (integer? 3.0) => false (float? 3.0) => true
(ratio? (/ 1 2)) => true (= 3 3.0) => false (== 3 3.0) => true
(+ 1 2) => 3 (exact) (/ 1.0 2) => 0.5 (double)
jolt= was already exactness-aware (values.ss) and == is value-equality, so
=/== match the JVM split. The reader preserves exactness (integer literals exact,
a/b ratios exact rationals, decimals/exponents flonums); backend_scheme emit-const
renders exact ints/ratios and flonums faithfully; the value-position arithmetic,
count, int, compare, bit ops, parseLong, string .length/.indexOf, range,
timestamps, and array bytes return exact integers (= JVM int/long) instead of
coercing to flonum. double/parseDouble/clojure.math floor|ceil|signum stay double.
Only the zero-Janet path carries the tower (the Janet reader loses exactness into
a double before emit). The prelude/all-flonum path is unaffected for compiled code;
the runtime reader is shared, so a couple of all-flonum reader assertions become
value (==) assertions. ~16 numeric corpus cases now give the JVM tower value vs the
Janet-era :expected and are allowlisted as tower divergences (Chez == reference
JVM) pending the corpus flip to JVM (jolt-ecz0). No BigDecimal type (1M).
Re-minted. zero-janet 2682 (floor 2698->2682, the reclassified tower cases), 0 new
divergences; fixpoint 10/10, bootstrap 6/6, spine 35/35, cli 49/49; Janet gate 155
files 0 failed.
Two Chez reader bugs, both JVM-parity gaps:
inc'/+'/foo' (trailing apostrophe) were mis-read as a symbol followed by a
quote macro, because the reader treated ' as a terminator. In Clojure ' is a
NON-terminating macro char (constituent after the first char). Since the seed
is minted on Chez, (def inc' inc) became (def inc 'inc), clobbering inc's var
cell with its own symbol -- so (var-get (var inc)) returned the symbol, not the
fn. Drop ' from the token terminator set; a leading ' still quotes.
^bytes [b] / ^String [x y] return-type hints: the Chez reader lowered ^meta on
a collection to a (with-meta vec meta) form, but emitted a QUALIFIED
clojure.core/with-meta while the Janet reader emits a bare with-meta -- so the
fn/defn macros' unwrap logic (matching the bare head) slipped past it and choked
on a non-vector arglist. Emit bare with-meta to match Janet, and unwrap a
(with-meta <vec> _) arglist in analyze-fn as a backstop.
Re-minted the seed. zero-janet 2699, prelude 2652, Janet gate 155/0, fixpoint
10/10, bootstrap 6/6, all 0 new divergences.
future/future-call run the body on a native thread (fork-thread) over the SAME
heap — JVM semantics, not Janet's isolated-heap snapshot. deref blocks on a
mutex+condition latch; timed (deref f ms val) uses an absolute deadline.
promise is a real blocking promise (deref parks until deliver), replacing the
Janet non-blocking atom shim. future?/future-done?/future-cancelled?/future-cancel
/realized? are native (the overlay versions read Janet map keys); re-asserted in
post-prelude over the overlay. pmap/pcalls/pvalues (overlay, over future) light
up for free.
Thread-safety this forces:
- atoms get a per-atom mutex; swap!/swap-vals! are a JVM-style CAS loop (f runs
outside the lock, so a watch/validator can deref the same atom); reset!/
compare-and-set! are atomic.
- the dynamic binding stack becomes a Chez thread-parameter, so each future/thread
has its own; Chez inherits it at fork, giving binding conveyance (the shim also
installs an explicit snapshot).
- Thread/sleep really sleeps now (a worker sleeping doesn't block the parent).
Re-minted the seed: future-call now resolves at compile time, so pmap compiles to
a var-deref instead of the host-static-call fallback that crashed. image.ss
unchanged.
Corpus: the 2 snapshot cases now match the JVM (shared) not Janet (isolated) —
allowlisted on both Chez gates; the two racy future-cancel cases allowlisted;
"promise undelivered" (blocks on JVM/Chez, profile :bucket :timeout) skipped like
:throws. Zero-Janet corpus 2544 -> 2569, 0 new divergences, floor raised. Full
Janet gate + JVM cert green.
jolt-byjr
The corpus certifier (test/conformance) flagged four cases where jolt's
hand-written :expected matched a real defect rather than Clojure. Fixed in the
jolt-core overlay, corrected the spec :expected, re-certified against JVM Clojure:
- ex-message: returns nil for a non-throwable (dropped the lenient string branch);
still returns the message for ex-info. (jolt-l8e8)
- munge: preserves the argument's type — a symbol munges to a symbol, not a string.
(jolt-hc35)
- print: (print nil) emits "nil", not "" (top-level nil guard; str yields "").
(jolt-pqio)
- bounded-count: uses the counted? fast path (full count), else counts up to n via
seq — was (min n (count coll)), wrong for counted colls. Added an uncounted-coll
spec case. (jolt-2507)
Removed the 4 :bug entries from known-divergences.edn (now certified), regenerated
corpus + profile, re-minted the Chez bootstrap seed (clojure.core changed). Gates:
Janet 155/0, JVM certify clean, both Chez corpus gates 2534 (floors raised),
bootstrap 6/6, fixpoint intact.
Makes the inc8 fixpoint the actual build. host/chez/bootstrap.ss loads a seed
(prelude, image) pair and rebuilds the clojure.core prelude + compiler image from
source via the on-Chez compiler — read/analyze/emit all on Chez, zero Janet.
The seed (host/chez/seed/{prelude,image}.ss) is the checked-in bootstrap
compiler, minted once via the fixpoint (driver/mint-chez-seed* iterates
bootstrap.ss from the Janet-emitted pair to a joint byte-fixpoint). It's a joint
fixpoint: rebuilding from an up-to-date seed reproduces it exactly. So a fresh
checkout + Chez (no Janet) yields a working jolt.
test/chez/bootstrap-test.janet spawns only chez, asserts the rebuilt artifacts
match the seed byte-for-byte and compile+run real cases. Drift (seed sources
changed) fails the test with a re-mint pointer; host/chez/seed/README documents
re-minting.
