An FFI library declares the system shared objects it binds in its deps.edn
(:jolt/native), with per-platform candidate sonames, :optional for feature-gated
deps, and :process for libraries that use the running process's own symbols
(libc sockets). jolt.deps collects them transitively; jolt loads them before the
library's namespaces are required, so foreign-fn bindings resolve — and a missing
required lib fails early with a clear message instead of a cryptic symbol error.
Replaces hardcoded soname-probing inside library .clj files.
The HTTP server moves out of the host into the jolt-lang/ring-janet-adapter
library, which binds sockets itself via jolt.ffi and shuts down cleanly. Drop
host/chez/http-server.ss and the obsolete ffi-server-test (the FFI collect-safe
path is covered by ffi-binding-test; the server by the adapter's own CI).
Derive os.name from Chez's machine-type (*osx -> Mac OS X, else Linux/Windows).
OS-branching code (socket sockaddr layout, etc.) needs the truth; a library
binding sockets via jolt.ffi reads os.name to pick the platform struct layout.
A library binding a blocking native call (accept/recv/connect/...) needs it
emitted __collect_safe so the thread deactivates for the call and doesn't pin
the stop-the-world collector. foreign-fn / defcfn take an optional trailing
:blocking; the backend emits (foreign-procedure __collect_safe ...). Needed for
the ring-janet-adapter socket-server port. ffi-binding-test asserts a thread
parked in a :blocking call doesn't block (collect).
The sqlite/jdbc functionality moves out of the host into the jolt-lang/db
library, which binds libsqlite3 (and libpq) itself via jolt.ffi. A baked
built-in jdbc.core would shadow the library's, so it's removed here. ring-app
gets jdbc.core from the db git dep instead.
A jolt library can now bind its own native dependencies and expose a Clojure API
over them — no jolt built-in required. This is the foundation for moving the
http-client / db / adapter functionality out of the host and into real libraries.
- jolt.ffi/foreign-fn (sugar: defcfn) is a compiler special form: a compile-time
-typed C signature lowers to a real Chez foreign-procedure (analyzer :ffi-fn ->
backend foreign-procedure), so calls are typed and marshaled, not eval'd.
- host/chez/ffi.ss provides the rest under jolt.ffi: load-library, alloc/free,
read/write/sizeof, ptr<->string, null/null?. Loaded after the loader snapshot
so a library's (require '[jolt.ffi]) still loads the macro side.
- Types: int/uint/long/ulong/int64/uint64/size_t/ssize_t/iptr/uptr/double/float/
pointer/string/void/uint8/char.
Validated end to end: a pure-Clojure file binds libc (getpid/strlen/abs) and
libsqlite3 (open/prepare/step/column/finalize over out-param pointers) and runs a
query. Gate test test/chez/ffi-binding-test.ss (make ffi); selfhost holds.
Three related namespace-resolution fixes surfaced porting the clojure.tools.logging
library, all general:
- chez-register-spec! treats a :use :only vector like :require :refer, so a
(:use [ns :only [names]]) clause actually brings those names in. Before, a bare
reference to an :only'd name resolved to nil.
- syntax-quote (hc-sq-symbol) qualifies a referred name to its SOURCE namespace,
not the compile namespace — so a macro that syntax-quotes a referred var (e.g.
clojure.tools.logging/spy reaching clojure.pprint's pprint) expands correctly.
- resolve consults :as aliases and :refers like ns-resolve does; (resolve
'alias/name) was returning nil because the alias wasn't expanded.
Bring the docs in line with the actual implementation now that Chez is the sole
substrate.
Deleted the migration/spike/handoff artifacts that only documented the Janet
era or the port effort: the port plan, phase-0 and foundational-runtime spike
writeups (+ the stray root-level copy), the self-hosting design notes, the
architecture-refactor plan, and spike/chez/RESULTS.md.
Rewrote the current reference docs against the Chez facts: building-and-deps and
tools-deps (no jpm/build step — bin/joltc off the checked-in seed, deps via
jolt.deps into ~/.jolt/gitlibs), libraries (SQLite is built-in jdbc.core over
libsqlite3, not a Janet driver), the conformance/spec test-flow docs (the Chez
corpus runner + certify, no .janet harnesses), and the transient / type-hint /
seed-overlay design notes (Chez representations: mutable transients, flat
copy-on-write vectors, HAMT maps, the seed/overlay twin). Fixed the README
collections line (vectors aren't 32-way tries) and added the ffi/transient gate
targets. rfc 0001's numerics open-question is resolved (the Scheme tower).
Renamed the built-in HTTP adapter to jolt.http.server only (dropped the
ring-janet.adapter alias — a Janet-era name).
The Chez port had landed transients as copy-on-write — each conj!/assoc!/etc.
rebuilt the whole persistent collection. Semantics were right but a transient
vector was O(n^2) to build (the persistent vector is a flat array, so every
conj! copied it); maps/sets were ~O(n log n) since the HAMT only path-copies.
This restores the Janet host's approach: true mutable backing, snapshot once on
persistent!.
vec : a growable Scheme vector (capacity + fill count); conj!/pop! amortized
O(1), persistent! hands off the buffer (exact fit) or trims once.
map : a Chez hashtable keyed by key-hash/jolt= (value equality, nil-safe);
persistent! folds it into a pmap.
set : a Chez hashtable; persistent! folds into a pset.
cow : fallback for anything else (e.g. a sorted coll) keeps the old
copy-on-write path, preserving jolt's superset.
get/count/contains?/nth see through each representation. Building a 400k vector
went from minutes (quadratic) to ~50ms (linear). assoc! keeps the variadic
dangling-key nil-pad on both vectors and maps. test/chez/transient-test.ss pins
the invariants and the linear-time property; wired in as `make transient`.
Two thread-safety bugs in the native FFI layer.
The HTTP server's accept/recv/send were plain foreign-procedures. A thread
inside a foreign call stays active for the stop-the-world collector, so the
accept loop sitting idle in accept() froze GC for the whole process whenever
another thread (a future, an async block) allocated. Mark the three blocking
calls __collect_safe so the thread deactivates for the call's duration —
collection proceeds while the accept thread waits. The args are an fd and
foreign-alloc'd buffers (outside the Scheme heap), so a collection mid-call has
nothing to move.
jolt.http-client built its -D header-file path from an unguarded (set! counter
(+ counter 1)) and counter mod 90000, with no per-process component. Concurrent
requests could compute the same path and clobber each other's headers. Use a
mutex-guarded monotonic counter plus the pid.
test/chez/ffi-server-test.ss exercises both (a (collect) while the server is
idle in accept(), temp-path uniqueness across threads, and a live request) and
is wired into the gate as `make ffi`.
malli loads and validates now. Three divergences surfaced building its registry
and :map schema:
dot-forms: (.iterator coll) on a map fell into the map-as-object branch and was
mis-read as a missing :iterator key (nil), so malli's -vmap got nil and crashed
on .hasNext. Route iterator through the collection-interop path — a jiterator
over the seq (the entry iterator for a map).
host-static: register clojure.lang.LazilyPersistentVector/createOwning (-vmap
fills an object-array then hands it over) and PersistentArrayMap/createWithCheck
(malli's eager entry parser relies on its duplicate-key throw; a missing class
was caught and mis-reported as ::duplicate-keys on every map schema).
Shaking the ring-app example's real library stack out against jolt surfaced a
batch of divergences from JVM Clojure, the biggest being evaluation order.
backend_scheme: call and recur arguments were emitted as bare Scheme operands,
so Chez's unspecified (right-to-left) order won out. Clojure evaluates left to
right, which selmer's reader loop relies on: (recur (add-node ... rdr) (read-char
rdr)) consumed a char early and dropped the first chars of every {{tag}}. Bind
operands to fresh temps in a let* (only when two or more can have side effects,
so hot calls over locals/consts stay un-wrapped). emit-ordered already did this
for collection literals; generalize it.
host-contract: syntax-quote now resolves the alias part of a qualified symbol
(impl/foo -> clojure.tools.logging.impl/foo) instead of leaving it bare, which
limped along via short-name matching until two loaded namespaces (reitit.impl,
clojure.tools.logging.impl) shared the short name and it broke.
collections: key-hash masks with bitwise-and, not fxand — jolt-hash is set!-
decorated per type (records return their own hash) and Chez's equal-hash can be a
bignum, so a key's hash isn't always a fixnum.
seq: even?/odd? handle bignums (JVM accepts any integer; the fxand crashed).
records: Keyword/Symbol .sym/.getName/.toString (honeysql's :clj branch reads
(.sym k)); Throwable .getMessage/.toString over a Chez condition.
host-static: __register-class-ctor!/__register-class-statics! so a host shim
(reitit.trie-jolt) can mirror a Java class.
natives-str: String.intern returns the string.
sqlite: jdbc.core fetch/fetch-one kebab-case column keys (the jolt-lang/db
convention; created_at -> :created-at).
io: a relative io/file path resolves against JOLT_PWD (the user's cwd), not the
repo root the launcher cd'd to — matches JVM cwd semantics, so config.edn loads.
cli: render an uncaught jolt throw (ex-info message + ex-data, or a condition)
instead of Chez's opaque "non-condition value" dump.
Gaps surfaced loading ring-core / hiccup / config / tools.logging on Chez:
- clojure.java.io/resource — resolve a named resource against the loader's
source roots (no classpath), returning a slurp-able File.
- (Object.) constructor — a fresh distinct value (lock / unique sentinel).
- *out* / *err* as dynamic vars over a port-writer, so (binding [*out* *err*] …)
and #'*out* compile and run (tools.logging, selmer).
- :refer :all now registers a refer-all relation, so an unqualified var from a
(require '[ns :refer :all]) resolves at compile time — including #'var.
- java.lang.Character interop ((.toString \+) etc.).
- StringReader accepts a char[] ((StringReader. (char-array s))), not just a
String.
- the \p{L} translation stops just below the UTF-16 surrogate gap (\x{D7FF})
instead of \x{10FFFF} — a range across the surrogates made irregex's char-set
construction call integer->char on a surrogate and crash.
sqlite.ss: jolt.sqlite + jdbc.core (jolt-lang/db's API) over the system
libsqlite3 — open/close, exec, a prepared query returning row maps, text/int/
double parameter binding, last_insert_rowid. The sqlite3 C API is non-variadic
so it binds directly.
http-server.ss: a minimal HTTP/1.1 server over BSD sockets (socket/bind/listen/
accept/recv/send via FFI), one connection at a time on a background accept
thread, synchronous Ring handlers. Parses the request line + headers + a
Content-Length body into a Ring request map (:body a StringReader), formats a
Ring response map back. Exposed as jolt.http.server and, for the example, as
ring-janet.adapter/run-server. macOS and Linux socket-option constants handled.
A synchronous HTTP client def-var!'d into jolt.http-client (get/post/put/delete/
head/request -> {:status :headers :body}), with :headers, :body, :query-params,
:content-type and :insecure?. It shells out to the system `curl` rather than a
direct libcurl FFI: on Apple Silicon curl_easy_setopt is variadic and Chez's
fixed-signature foreign-procedure can't place the value arg on the stack where
the ABI expects it, so a direct bind silently drops the option. curl gives the
same native TLS/redirect/gzip with no per-platform C shim.
format now honours width and the -/0 flags (%-30s, %5d, %05d), not just %.Nf
precision — it was emitting the directive literally.
A minimal truecolor PNG encoder over Chez bytevectors — CRC-32 / Adler-32
framing with DEFLATE "stored" blocks, so there's no compressor to carry. Restores
the jolt.png built-in (image/put!/write) the old host provided; def-var!'d into
the jolt.png namespace and loaded in the CLI before the loader's baked-namespace
snapshot, so (require '[jolt.png]) resolves with no source file. Output verified
to decode as a valid PNG (signature, IHDR, CRC-correct chunks, inflatable IDAT).
A mutable Map keyed by jolt values (jolt-hash/=) with put/get/putAll/containsKey/
size/remove/keySet/values/entrySet — enough for libraries that build a fast
lookup table (malli's fast-registry doto's a HashMap then .get's it). The
"No method M for value" dot-dispatch error now includes the value, which makes
a wrong-type interop target far easier to pin down.
Reader / loader:
- #?@ splicing reader conditionals now actually splice the matched collection's
items into the enclosing sequence; the splice flag was read but ignored, so a
binding vector like [a #?@(:clj [b (.foo b)])] lost its alignment.
- the file loader reads by position and skips a top-level form that reads as
nothing (a :cljs-only #?, a #_ discard, a trailing comment) instead of
treating it as EOF — which silently dropped the rest of a large .cljc file.
- jolt's reader feature set now includes :clj (was {:jolt :default}). jolt is a
Clojure/JVM-compatible host that emulates clojure.lang.* and java.* interop,
so it reads the :clj branch of a .cljc library, not :cljs. This also lets four
more reader-conditional corpus cases pass (floor 2726 -> 2730).
Backend:
- munge-name escapes ' (prime) -> _PRIME_; a Clojure symbol like f' otherwise
emitted a bare ' into Scheme, which is the quote reader macro and unbalanced
the output.
Host shims:
- clojure.java.io/writer (pass through a StringWriter, file-back a path) and a
readLine on the string reader, so line-seq over (io/reader …) works (markdown).
A better "unsupported destructuring pattern: <pat>" error message.
joltc grew from a single -e expression into a real project runner. require now
loads a namespace's .clj/.cljc from the source roots transitively (load-once),
so a multi-file project works; the corpus/unit gates load compile-eval.ss but
not the loader, so their alias-only require is unchanged.
jolt.deps resolves a deps.edn into ordered source roots — git + local deps only
(no Maven), breadth-first so a top-level pin wins, with aliases (:extra-paths/
:extra-deps/:main-opts) and tasks. Git deps clone into a sha-immutable cache
($JOLT_GITLIBS, else ~/.jolt/gitlibs) by shelling out to git via a new
jolt.host/sh primitive. jolt.main dispatches run -m / -M:alias / -A / repl /
path / a deps.edn task. The launcher passes the user's cwd as JOLT_PWD (the
project dir) since it cd's to the repo root for the runtime's relative loads.
Four runtime/reader gaps that blocked real libraries (hiccup, commonmark):
- reader: a type hint on a code form (^String (to-str x)) was lowered to a
runtime (with-meta (to-str x) {:tag String}), mis-applying the hint to the
call's RESULT and throwing when it's a string/number. A :tag hint on an
evaluated form is compile-time only in Clojure — attach it to the form
instead. Collection literals (^:foo [1 2 3]) still get runtime metadata.
- deftype: register the ctor globally by simple class name (like StringBuilder)
so (Name. ...) interop resolves ns-agnostically. host-new resolved the ctor
against the runtime current ns, which is the caller's, not the defining ns,
once a deftype is used across files.
- protocol dispatch: canonical-host-tag now strips the clojure.lang. prefix too
(clojure.lang.Keyword -> Keyword), and keywords/symbols carry the Named tag,
numbers a Ratio tag. An (extend-protocol P clojure.lang.Keyword ...) was
missing the dispatch, so e.g. hiccup rendered <:html> instead of <html>.
- regex: parse leading Java inline flags ((?s)/(?i)/(?m)) and pass the
equivalent irregex options (single-line/case-insensitive/multi-line); irregex
rejects the inline syntax. Adds a java.util.Iterator shim ((.iterator coll)/
.hasNext/.next) for the run!-style loop hiccup compiles.
The two future-cancel cases — (future-cancel (future 1)) and the
future-cancelled? variant — depend on whether future-cancel catches a
trivial future in-flight, which is pure thread-scheduling luck. They were
allowlisted but still counted toward `pass` whenever the race resolved
favorably, so the floor (2728) silently assumed both passed. On a fast dev
machine they always pass; on CI's loaded shared runner one races to a
divergence, dropping pass to 2727 and failing the gate.
Skip them like the undelivered-promise case (neither pass nor fail) so the
race can't perturb the count. Floor drops to the deterministic 2726.
deftype fields tagged ^:unsynchronized-mutable / ^:volatile-mutable can now be
reassigned in place from a method, as on the JVM. A jrec stores fields as cons
cells, so a new jolt-set-field! mutates the pair with set-cdr!. The deftype macro
rewrites (set! mutable-field v) in a method body to (set! (.-field inst) v), and
the analyzer compiles a (set! (.-field obj) v) target to jolt-set-field! — so
both the rewritten symbol form and an explicit interop (set! (.-root this) v) go
through one path. Field reads remain a snapshot at method entry, which is correct
for the universal read-then-set pattern (a repeated set! of the same field in one
call would read the entry value).
Closes the set!-of-local SCI failures: SCI load 202 -> 205/218.
Bring the formal definition in line with this session's language work:
- grammar.ebnf: numbers are a real tower (exact integer / Ratio / double); the M
suffix reads a real BigDecimal, N an exact integer (drop the stale Janet note).
- 02-reader S5: M is a real java.math.BigDecimal with scale-insensitive equality.
- 03-special-forms: document the read -> macroexpand -> analyze order (macros
expand before special-form dispatch); special-form heads are not shadowable but
macros are and value-position locals may be named like a special; set! on a var
sets the innermost binding (else root); letfn is a primitive with letrec*
semantics.
Found in a read/eval review: a local named like a special form wrongly took
over operator position. (let [if (fn ...)] (if true 1 2)) returned the fn, but
per spec section 3 (and the reference) special-form heads are not shadowable;
only macros are. Two fixes: drop the (not shadowed) guard on the special-form
branch of analyze-list (so an (if ...) head is always the special), and prefix
a local whose name is a Scheme keyword when emitting (so a value local legally
named if does not shadow the (if ...) the back end emits). Value-position
locals named if/or/case still work.
The analyzer checked special forms before expanding macros, the reverse of the
canonical read -> macroexpand -> analyze order (Clojure/CLJS analyze-seq). Move
macroexpansion to the front of analyze-list. Knock-on fixes:
- letfn was both a (broken) macro expanding to let* AND a primitive special
(analyze-letfn, proper letrec*). Macroexpand-first surfaced the macro, breaking
mutual recursion; remove the macro, keep letfn a primitive.
- defmacro is now compiled by the analyzer (a :set-var-style :defmacro node that
defs the expander fn via the fn macro — so destructuring arglists desugar — and
marks the var a macro), so a non-top-level (when … (defmacro …)) works. The
runtime spine's separate top-level defmacro interception is removed: one path.
SCI load 162 -> 202/218.
The analyzer punted set! as uncompilable. Add it as a special form: (set! sym
val) on a var emits jolt-var-set, which updates the innermost thread binding (or
the root when unbound), returning val. A local target (deftype mutable field) or
an interop (.-field) target stays uncompilable for now. Also defines
*warn-on-reflection* (false) so set! on it resolves. SCI load 186 -> 196/218.
Chez rejects duplicate lambda formals, so any (fn [_ _] ...) failed to compile
— including every macro expander, whose &form/&env slots both expand to _. The
analyzer now renames each earlier duplicate param to a fresh name (Clojure binds
the last occurrence, so the earlier ones are shadowed and unreferenceable). SCI
load 162 -> 186/218.
A vector's seq is now a real chunked-seq (chunked-seq? true), matching Clojure/
CLJS. Each vector-seq cell carries its backing vector + element index as two
cseq fields (cvec/ci, no extra allocation vs the old lazy cell), so:
- chunk-first hands out a 32-element block (a pvec slice), chunk-rest is the
seq at the next block boundary — the ChunkedSeq contract (chunk-first ++
chunk-rest == the seq);
- reduce/transduce take a fast path that walks the backing vector by index in
a tight loop with no per-element seq cells (reduce over a 1M-vector ~0.4s).
The seq cell stays a cseq, so first/rest/count/printing and the ~26 cseq?
dispatch sites are untouched. The eager chunk-buffer model (chunk-buffer/chunk/
chunk-cons) is preserved for the round-trip case. No seed change (runtime only).
(type r) returned a symbol user.TyR, so (= (symbol (str (type r))) (type r))
was true; the JVM's type is a Class (not a Symbol) so it's false. jolt models
classes as strings, so a record's type is now its ns-qualified class-name
string — equal to (class r), as on the JVM where type and class coincide for a
record. The symbol-keyed print-method defmethods already fall through to the
default record printing, so they're unaffected. Closes type-of-record.
bigdec / 1.5M / 0.0M silently produced doubles. Add a jbigdec value type
{unscaled, scale} over Chez exact integers (host/chez/bigdec.ss): value =
unscaled * 10^-scale. An M-suffix literal reads to a :bigdec form that the back
end lowers to jolt-bigdec-from-string (same IR-leaf path as #inst/#uuid); bigdec
coerces a number/string. Equality is by value (1.0M = 1.00M true, 3M = 3 false),
str drops the M and pr keeps it, class is java.math.BigDecimal, decimal? is true.
Arithmetic contagion isn't modelled (out of scope). The old corpus cases passed
spuriously as doubles; they now exercise a genuine BigDecimal.
(def ^String tv ...) left (:tag (meta (var tv))) as the unresolved "String";
the JVM compiler resolves the hint to java.lang.String at def time. Add a
resolve-class-hint host seam (built from the existing class-token table) and
resolve a def's :tag through it in the analyzer. The reader path
(read-string "^String x") stays unresolved, matching the JVM (only the
compiler resolves). Closes ^Type-tag-on-var.
(map? *in*) was true because *in* was a plain map of read-line-fn/read-fn
closures; the JVM *in* is a java.io.Reader so map? is false. A defrecord
doesn't help (records are maps). Make the reader a reify over a new IReader
protocol — a non-map value — and route read/read-line/read+string/line-seq
through its -read-line/-read-form/-read+string methods instead of keyword
access. with-in-str's __string-reader and the stdin *in* both reify it.
Closes *in*-bound + *in*-is-bound.
A macro like (defmacro cur-ns [] `(str ~*ns*)) splices the live *ns* value
into its expansion, leaving an opaque jns object as a list element. The
analyzer had no way to carry a runtime value and threw uncompilable — the last
remaining corpus crash. Recognize a jns via the host contract (form-ns-value?)
and emit a :the-ns leaf that reconstructs it by name (intern-ns!) at the call
site, the same IR-leaf pattern as regex/inst/uuid. Closes unquote-*ns*-in-
template; corpus crash count -> 0.
A namespace fast path rather than a general constant pool: it's the only
embedded-value case in the corpus and the common real-world one (libs splice
~*ns*). A general pool can come later if other value types appear.
deftype/defrecord inline protocol methods went through extend-type ->
register-method, so a record implementing a protocol inline showed up in
(extenders P) — the JVM only lists extend/extend-type/extend-protocol
registrations there (inline impls compile into the class). Add
register-inline-method: it registers for dispatch under the record tag but
skips the extender mark. The mark lives inside type-registry so the per-case
corpus prune restores it. Closes corpus lists-extended-type + seq-of-tags.
The defmethod/print-method record cases, symbol-hint, and source-order entries no
longer diverge (closed by the defmethod-setup + earlier fixes); drop them. 0 new
divergences, corpus 2720/2741, 20 genuine gaps remain.
definterface now expands to (do (def name {}) 'name) so (var? (definterface ...))
is false, matching the JVM where it yields the interface Class. ns-imports returns
the 96 auto-imported java.lang classes (short symbol -> canonical name) so
(count (ns-imports 'user)) is 96. Re-minted for the macro change. Corpus 2718->2720.
class number/string/keyword/name + atom?/instance? Atom pass after the class
refinements; drop their stale allowlist entries. Corpus 2705/2741 0 new div.
The prior fix resolved an unqualified defmethod to clojure.core's multifn, which
broke SCI (it relies on per-ns shadow multimethods — hung loading core_protocols).
Keep the shadow, but when auto-creating it copy the dispatch fn + default from a
same-named clojure.core multifn (e.g. print-method's 2-arg dispatch) instead of
the 1-arg identity that crashed (print-method x w). Also trim the FQN class
tokens to value classes only (the collection interfaces shadowed names SCI uses).
Corpus 2705/2741 0 new div; SCI 162/218 restored; cross-ns + direct print-method
overrides work.
