The host/chez directory mixed jolt's own runtime (value model, seq, reader,
vars, ns, multimethods) with the shims that emulate the JVM: java.* / javax.*
classes, clojure.lang interfaces, and the host-class registry they hang off.
Move that JVM-emulation layer into host/chez/java/ so it reads as a distinct
unit instead of being interleaved with the platform runtime.
Moved (content unchanged): host-static, host-static-methods,
host-static-classes, host-class, dot-forms, records-interop, byte-buffer,
io, io-streams, inst-time, java-time, bigdec, natives-queue, natives-str,
natives-array, math, concurrency, async, ffi.
The load paths in rt.ss/cli.ss and the build.ss runtime manifest are updated
to point at java/; the build inliner follows the (load ...) strings, so the
AOT path needs no other change. All runtime shims, no seed source touched
(the three .clj edits are doc comments), so no re-mint.
Gate green: make test (selfhost fixpoint, certify 0-new, sci 211, infer),
shakesmoke (4 apps byte-identical).
A post-conformance review (chiasmus) flagged fresh-sym defined byte-identically
in 00-syntax and 30-macros; 00-syntax loads first, so the second is redundant.
Also note why deftype uses group-by-head while extend-protocol uses
parse-extend-impls (the latter must treat a computed class type in head position).
No behavior change.
Conformance gaps surfaced re-running the library suites:
- defn now keeps a leading docstring as :doc metadata — it was dropped, so
(:doc (meta #'f)) was always nil. Rides the def docstring slot.
- assert (and :pre/:post) throw a real AssertionError instead of an ex-info, so
(catch AssertionError …) / (thrown? AssertionError …) match, with Clojure's
"Assert failed: <msg>\n<form>" message.
- instance? clojure.lang.Seqable was conflated with ISeq, so a vector/map read
as not-Seqable. Split them: Seqable covers every persistent collection, ISeq
only seqs.
Running cognitect aws-api's pure test namespaces (signing/shapes/protocols/
util/retry/endpoints) surfaced general gaps:
- extend-protocol/extend-type accept a computed class type, e.g.
(Class/forName "[B") for the byte-array class — the byte-array idiom data.json
and aws-api use. The macro grouping handled only symbol/nil heads (it crashed on
a list type); type->name resolves a Class value via .getName; a byte-array
dispatches on the "[B" host tag.
- java.nio.ByteBuffer over a jolt byte-array (wrap/allocate/get/put/array/
remaining/position/limit/duplicate/flip), plus extend-protocol to it.
- java.util.Arrays (equals/copyOf/copyOfRange/fill) and java.util.Random
(nextBytes/nextInt/…).
- java.net.URI/create and clojure.lang.RT/baseLoader statics.
- clojure.core.async/promise-chan (deliver-once, peek-don't-pop).
- a failed java.time parse throws DateTimeParseException (typed), so
(catch DateTimeParseException …) matches it instead of leaking an untyped
condition.
The XML side lives in the jolt-lang/xml library (libxml2 over jolt.ffi); ByteBuffer
stays in core as a generic java.nio primitive.
Gate: make test green (corpus +6 JVM-certified rows, 0 NEW divergence; unit
553/553; SCI 211).
Shaking out clojure.core.memoize (207 assertions, 0 fail) cleared several
general gaps:
- deref/@ on a deftype or reify implementing clojure.lang.IDeref dispatches to
its deref method (RetryingDelay / make-derefable).
- deftype mutable fields (^:unsynchronized-mutable / ^:volatile-mutable) are
read live: a set! within a method is observed by a later read in the same
invocation, not the entry-time capture. Needed for double-checked locking.
Immutable fields stay let-bound. Field reads rewrite to (.-field inst) with
lexical-shadow tracking.
- def metadata values are evaluated, like Clojure: ^{:k (f)} stores (f)'s
result and ^{:af some-fn} the fn. :tag stays a literal hint.
- try dispatches catch clauses by class in order via the exception supertype
hierarchy; a non-matching value re-throws, an untyped host condition is caught
by a RuntimeException/Exception/Throwable clause. Previously the last clause
won and the class was ignored.
- locking takes a real per-object monitor (recursive Chez mutex) now that
futures/agents/threads share one heap; it was a no-op.
- supers/ancestors reflect a small modeled JVM interface hierarchy, so
(ancestors (class f)) yields Runnable/Callable (core.memoize's arg check).
- AssertionError / Error constructors.
JOLT_FEATURES is gone from the docs: it isn't read anywhere on Chez, and the
reader already includes :clj in its default feature set. RFC 0002's
{:jolt :default} design was reverted in the reader; docs now match the code.
Raises the SCI floor 205 -> 210.
Further clojure.core.cache fixes (198 -> 257 of its assertions):
- delay: a throwing body re-ran on every force and never became realized?. Run it
once like Clojure's Delay — cache the exception, mark realized, re-throw the same
on each deref. Fixes value-fn memoization / cache-stampede protection.
- deftype/defrecord: a method name appearing in two protocols with different
arities (data.priority-map's seq is in IPersistentMap [this] AND Sorted
[this asc]) registered per-protocol and shadowed; merge clauses by name across
all protocols into one multi-arity fn.
- empty?/peek/pop (IPersistentStack) dispatch through a deftype's methods; (= a-
deftype other) uses its equiv method (so caches compare to their backing map);
seq handles a host iterator (iterator-seq over .iterator).
- pop of an empty PersistentQueue returns it, like the JVM (was an error).
JVM-certified corpus rows. make test + shakesmoke green.
General fixes shaken out running clojure.core.cache (66 -> 198 of its assertions):
- Map destructuring applied an :or default only for :keys/:strs/:syms, not a
direct {x :x} binding — so {x :x :or {x 9}} (and the & {…} kwargs form) ignored
the default. Apply it for the direct binding too.
- fn didn't implement :pre/:post: a leading conditions map was evaluated as a body
literal (so % was unbound and (.q %) blew up). Recognize it and assert pre
before the body, bind % to the result, assert post, return %.
- (.q inst) on a deftype field with no matching method reads the field, like the
JVM (was "No method q").
- A deftype implementing the clojure.lang collection interfaces now dispatches
dissoc (without), contains? (containsKey), peek/pop (IPersistentStack), and
keys/vals (via its Seqable seq) through its methods — they were field-only, so
core.cache's caches and data.priority-map didn't behave as maps.
JVM-certified corpus rows for each. make test + shakesmoke green.
Finishes core.match — its full test suite (115/115) now passes, including the
two patterns the earlier work left out:
- Regex-literal patterns. A #"…" now reads as a regex VALUE (Clojure parity: the
reader constructs the Pattern, so a macro receives a regex, not jolt's tagged
form), and the analyzer compiles a regex value to the same :regex IR leaf via
its source. emit-quoted handles a quoted regex; a regex value carries the
java.util.regex.Pattern host tag so extend-protocol/instance? dispatch on it.
- Primitive-array patterns. A ^Type hint's :tag is now the SYMBOL (e.g. `ints`),
matching the JVM, so core.match's array-tag lookup engages the array
specialization (alength/aget). jolt's :tag consumers already tolerate a symbol
(hc-cell-num-ret normalizes; tag->nkind/def-meta handle both).
Also: a library-conformance directive in CLAUDE.md, and the supported-libraries
list (docs + site) simplified to one-line entries — a listed library is assumed
to work fully, so no tallies or feature enumerations. core.match + transit-jolt
added to the list.
Seed change (reader/backend/30-macros) -> re-minted; the rest runtime. JVM-
certified corpus rows; the stale `symbol hint -> :tag` divergence is dropped from
the allowlist (jolt now matches the JVM). make test + shakesmoke green.
Running clojure.core.match (a macro-heavy library that builds its compiler out
of deftypes implementing clojure.lang interfaces) shook out a cluster of general
gaps. Its own suite goes from not-loading to 111/115 assertions.
- deftype/defrecord implementing a clojure.lang collection interface now drives
the core fns: Indexed -> nth, Counted -> count, Associative -> assoc, ILookup
-> get/valAt (non-field keys only, so a method's own field bindings don't
recurse), ISeq -> seq/first/rest, IPersistentCollection -> conj, IFn -> the
value is callable. A jrec is still a map of fields by default; the interface
method wins when declared.
- Multi-arity inline methods are grouped into one fn (a type with (nth [_ i]) and
(nth [_ i x]) kept only the last before). Built as data, not a nested
syntax-quote, so a `(= ~ocr ~l) method body keeps its unquotes.
- instance?/satisfies? recognize a protocol a type implements, including a MARKER
protocol with no methods (core.match's IPseudoPattern) — deftype/defrecord now
record protocol satisfaction even with zero methods. Added ILookup/Indexed/
Counted to the instance? taxonomy for the built-in collections.
- Syntax-quote: a fully-qualified class name (clojure.lang.ILookup) stays bare
instead of being namespace-qualified; (unquote x) is detected in a lazy seq
(a macro that builds its template with map, e.g. deftype's rewrite-set).
- clojure.set union/intersection/difference are variadic (& sets) + union 0-arity.
- java map view methods: keySet/values/entrySet/size/isEmpty.
- deprecated java.util.Date getters (getYear/getMonth/...) + the multi-arg
(Date. year-1900 month0 date hrs min) constructor.
Seed change (deftype/defrecord macros + clojure.set) -> re-minted; the rest are
runtime. 11 JVM-certified corpus rows; make test + shakesmoke green.
Four general gaps, shaken out by loading clojure.spec.alpha:
- Special forms were shadowable by a same-named macro. analyze-list
macroexpanded before checking special forms, so a ns that redefs def/and/or
(spec excludes them via :refer-clojure :exclude) made a bare def resolve to
the macro instead of the special form, breaking every defn after. Now a head
in the special-form set is never macroexpanded, matching the reference
macroexpand1 isSpecial check.
- reify dropped all but the last arity of a multi-arity protocol method (spec
reifies (specize* [s]) and (specize* [s _])). The macro keyed methods by name
and overwrote; now it groups arities into one multi-arity fn.
- reify instances did not implement IObj: with-meta threw and (instance?
clojure.lang.IObj r) was false. Every Clojure reify carries metadata. with-meta
now copies the reify to a fresh identity (shared method table) and keys its
meta; instance? IObj/IMeta is true for any reify. This was the registry bug —
spec's with-name returned nil for specs, so get-spec missed.
- (set! (. Class field) val) was rejected. spec toggles
clojure.lang.RT/checkSpecAsserts this way; the analyzer now lowers it to a
jolt.host/set-static-field! call over a mutable-statics table, and a plain
Class/field read consults that table.
Also: .name/.getName on a Namespace and .ns/.sym on a Var (spec's ns-qualify /
->sym). analyzer + reify are seed sources (re-minted). spec.alpha now does
valid?/conform/cat/keys/explain-str/check-asserts. tick.alpha.interval-test still
needs time-literals data readers (separate).
ZoneOffset/ZoneId (SHORT_IDS, fixed-offset + UTC + system; named zones via a
fixed-offset table), ZonedDateTime/OffsetDateTime/OffsetTime, Clock (fixed/
system, with now [clock] arity), and DateTimeFormatter integration (ofPattern
+ ISO_* constants, .format/.parse over the rich java.time values via the
inst-time.ss pattern engine). systemDefault resolves to UTC to keep the
#inst atZone/toInstant round-trip machine-tz-independent.
tick.core + tick.protocols + tick.locale-en-us load; tick's api_test runs
31 tests / 352 pass / 7 fail / 0 error. The 7 are host gaps: named-zone DST
(no tzdb), French locale month names (no locale DB), nanosecond Instant.
General fixes surfaced by tick: :ns/keys map destructuring ({:tick/keys [..]})
in 00-syntax.clj (re-minted), and extend-protocol to java.time classes
(records.ss host-type-set). 12 corpus rows certified vs JVM. make test +
shakesmoke green, selfhost holds, 0 new divergences, data.json stays 138/139.
A slash-free dotted symbol with a Capitalized final segment (java.util.Map,
clojure.lang.Named, java.time.Instant) now self-evaluates to its name string
instead of resolving to nil — jolt models a class as its name, so a library
can extend a protocol to, or instance?-check, a host class jolt has no shim
for. hc-resolve-global classifies these as :class; the analyzer emits a const.
extends? now matches when either the query or the registered tag is a dotted
suffix of the other, so (extends? P java.util.Collection) finds the impl
extend registered under the canonical short tag.
Add DateTimeFormatter/ISO_INSTANT (UTC, trailing Z).
These unblock loading clojure.data.json, which dispatches JSONWriter on
java.util.Map/Collection/CharSequence/Instant and defaults a formatter to
ISO_INSTANT.
read-string/read now return real sets for #{...} literals (top-level and
nested) instead of the reader's {:jolt/type :jolt/set} form — the data
seams convert set forms to sets (recursing, preserving metadata and source
map-key order); clojure.edn already did this. The compiler keeps reading
via the raw reader, so set literals in code stay forms the analyzer lowers.
format %x now emits lowercase hex (Chez number->string is uppercase); %X
unchanged.
extend and extends? handle a nil target type (host tag "nil"), matching
extend-type — protocols can be extended to nil via the function form, not
just the macro.
Found porting transit/data.json and shaking out aero.
conj/assoc/dissoc/disj/pop/into and empty now thread the receiver's
metadata onto the result, matching Clojure (each op constructs a new
collection with meta() carried forward; coll.empty() is
EMPTY.withMeta(meta())). The metadata side-table is now weak so meta on
intermediate collections is reclaimed with them, and empty-list-t carries
an (unused) field so a metadata-bearing () is a distinct identity from the
shared singleton instead of leaking meta onto every ().
Unblocks metadata-driven walks (aero/integrant): (into (empty form) ...)
now preserves a vector/map/set's metadata, so a postwalk whose outer fn
reads (meta x) sees it.
The 1123-line collection tier is the largest source file. Cut it at two existing
section banners into 20-coll (predicates, printing, hierarchies, pure-over-core
leaves), 21-coll (rand/sort seams, the test runner, fn combinators), and 22-coll
(canonical Clojure ports, transduce/into, JVM-shape stubs). No macros in this tier,
so order is the only constraint; the emit-image manifest lists the three in
sequence. Re-minted seed is identical apart from gensym label renumbering.
- take-last / drop-last return seqs, not vectors: take-last wraps in seq; drop-last
is the JVM (map (fn [x _] x) coll (drop n coll)) form (lazy, () when empty).
- cycle is lazy ((lazy-seq (concat coll (cycle coll)))) so it no longer counts its
argument and terminates on a lazy/infinite input.
- fold's foldable-call catch uses :default, matching the rest of jolt-core and
also catching a raw host condition from a folding primitive.
- alts! rejects non-channel ports with a clear error (put specs / :default are
unsupported) instead of crashing inside ac-poll!.
- Misc: drop the unreachable second getCause clause; jolt-nth on a string raises
'nth "index out of bounds" like the vector branch; name the inline fixpoint cap;
bld-sh-capture rejoins lines with newlines; clarify a couple of comments.
Rename src/jolt -> stdlib (the runtime-loaded layer; jolt-core stays the
seed-baked layer) and update the loader / emit-image / doc paths. Drop dead
code: the spike/ experiments, the duplicate clojuredocs-export.edn (json moves
to tools/), the Janet-era jolt.http binding, and the orphaned
persistent_vector.clj whose ns/path didn't even match.
Strip porting residue from comments and docstrings across host/chez, jolt-core,
stdlib, tests, and docs: internal issue ids, "Phase N" markers, and the "vs
Janet" historical exposition, leaving present-tense descriptions and the real
JVM-Clojure semantic contrasts. Same pass over the corpus suite labels. The seed
is unchanged (docstrings/comments aren't emitted), so the self-host fixpoint and
corpus are untouched.
Port tools/spec_coverage.py off the dead janet probe to bin/joltc and regenerate
coverage.md; drop the dead :host/janet rule from certify.clj and regenerate the
conformance profile. Add docs/host-interop.md (the JVM shims and how to register
your own host class from a library) and a writing-style note in CLAUDE.md.
Stabilize the four racy concurrency corpus cases (future-cancel and agent
send/send-off): give the future a sleeping body and the agent a slow action, so
cancel reliably catches an in-flight future and deref reliably reads the
pre-update snapshot. They certify deterministically now, so drop their :flaky
allowlist entries and the orphaned legend.
Reader / loader:
- #?@ splicing reader conditionals now actually splice the matched collection's
items into the enclosing sequence; the splice flag was read but ignored, so a
binding vector like [a #?@(:clj [b (.foo b)])] lost its alignment.
- the file loader reads by position and skips a top-level form that reads as
nothing (a :cljs-only #?, a #_ discard, a trailing comment) instead of
treating it as EOF — which silently dropped the rest of a large .cljc file.
- jolt's reader feature set now includes :clj (was {:jolt :default}). jolt is a
Clojure/JVM-compatible host that emulates clojure.lang.* and java.* interop,
so it reads the :clj branch of a .cljc library, not :cljs. This also lets four
more reader-conditional corpus cases pass (floor 2726 -> 2730).
Backend:
- munge-name escapes ' (prime) -> _PRIME_; a Clojure symbol like f' otherwise
emitted a bare ' into Scheme, which is the quote reader macro and unbalanced
the output.
Host shims:
- clojure.java.io/writer (pass through a StringWriter, file-back a path) and a
readLine on the string reader, so line-seq over (io/reader …) works (markdown).
A better "unsupported destructuring pattern: <pat>" error message.
deftype fields tagged ^:unsynchronized-mutable / ^:volatile-mutable can now be
reassigned in place from a method, as on the JVM. A jrec stores fields as cons
cells, so a new jolt-set-field! mutates the pair with set-cdr!. The deftype macro
rewrites (set! mutable-field v) in a method body to (set! (.-field inst) v), and
the analyzer compiles a (set! (.-field obj) v) target to jolt-set-field! — so
both the rewritten symbol form and an explicit interop (set! (.-root this) v) go
through one path. Field reads remain a snapshot at method entry, which is correct
for the universal read-then-set pattern (a repeated set! of the same field in one
call would read the entry value).
Closes the set!-of-local SCI failures: SCI load 202 -> 205/218.
The analyzer checked special forms before expanding macros, the reverse of the
canonical read -> macroexpand -> analyze order (Clojure/CLJS analyze-seq). Move
macroexpansion to the front of analyze-list. Knock-on fixes:
- letfn was both a (broken) macro expanding to let* AND a primitive special
(analyze-letfn, proper letrec*). Macroexpand-first surfaced the macro, breaking
mutual recursion; remove the macro, keep letfn a primitive.
- defmacro is now compiled by the analyzer (a :set-var-style :defmacro node that
defs the expander fn via the fn macro — so destructuring arglists desugar — and
marks the var a macro), so a non-top-level (when … (defmacro …)) works. The
runtime spine's separate top-level defmacro interception is removed: one path.
SCI load 162 -> 202/218.
bigdec / 1.5M / 0.0M silently produced doubles. Add a jbigdec value type
{unscaled, scale} over Chez exact integers (host/chez/bigdec.ss): value =
unscaled * 10^-scale. An M-suffix literal reads to a :bigdec form that the back
end lowers to jolt-bigdec-from-string (same IR-leaf path as #inst/#uuid); bigdec
coerces a number/string. Equality is by value (1.0M = 1.00M true, 3M = 3 false),
str drops the M and pr keeps it, class is java.math.BigDecimal, decimal? is true.
Arithmetic contagion isn't modelled (out of scope). The old corpus cases passed
spuriously as doubles; they now exercise a genuine BigDecimal.
(map? *in*) was true because *in* was a plain map of read-line-fn/read-fn
closures; the JVM *in* is a java.io.Reader so map? is false. A defrecord
doesn't help (records are maps). Make the reader a reify over a new IReader
protocol — a non-map value — and route read/read-line/read+string/line-seq
through its -read-line/-read-form/-read+string methods instead of keyword
access. with-in-str's __string-reader and the stdin *in* both reify it.
Closes *in*-bound + *in*-is-bound.
deftype/defrecord inline protocol methods went through extend-type ->
register-method, so a record implementing a protocol inline showed up in
(extenders P) — the JVM only lists extend/extend-type/extend-protocol
registrations there (inline impls compile into the class). Add
register-inline-method: it registers for dispatch under the record tag but
skips the extender mark. The mark lives inside type-registry so the per-case
corpus prune restores it. Closes corpus lists-extended-type + seq-of-tags.
definterface now expands to (do (def name {}) 'name) so (var? (definterface ...))
is false, matching the JVM where it yields the interface Class. ns-imports returns
the 96 auto-imported java.lang classes (short symbol -> canonical name) so
(count (ns-imports 'user)) is 96. Re-minted for the macro change. Corpus 2718->2720.
Allowlist review found three addressable divergences:
- unchecked-char returned a number; the JVM returns a char.
- the readable printer (pr-str, coll elements, the -e/REPL printer) rendered
infinities as Infinity/inf; Clojure's readable form is ##Inf/##-Inf/##NaN
(str/print still gives Infinity). So (pr-str ##Inf) => ##Inf, (str [##Inf]) =>
[##Inf], (str ##Inf) => Infinity.
Corpus 2695->2698; allowlist 43->40 (drop the 3 now-passing entries). Re-minted.
sorted-map seq/first/entries built plain [k v] vectors, so map-entry? was false
and key/val threw. Build them via a new jolt.host/map-entry seam (entry-flagged
pvec), matching a regular map's entries. Re-minted.
(into [] (dedupe) coll) / (sequence (dedupe) coll) threw an arity error — dedupe
was [coll]-only. Add the 0-arg stateful transducer (tracks [seen? prev] in a
volatile, no sentinel). Re-minted.
- == 1-arg returns true for any value (Clojure short-circuits before the number
check), not 'requires numbers'.
- current-time-ms wired to now-millis so the time macro works.
- subvec truncates float/ratio indices via long (Scheme quotient rejects flonums).
- defonce checks bound? not var-get — in a top-level do the name is already an
unbound interned cell, which var-get throws on.
- drop the line-seq corpus row (used janet/spit, N/A); allowlist char-array
(needs Class/forName "[C").
Corpus 2678->2683, floor raised. Re-minted. Full gate green; CI green.
jolt-cf1q.7
Rephrase comments that pointed at deleted Janet files (emit.janet, the seed
sources, 'the Janet back end punts ...') to present-tense descriptions of the
Chez behavior. Comment/docstring-only; the self-host fixpoint is unchanged
(comments don't affect the compiled seed).
Delete five files that were Janet-host shims with no Chez path: clojure.java.io
(provided natively by host/chez/io.ss), and jolt.{nrepl,png,interop,shell}
(the janet.* bridge, os/shell, janet.net — none exist on Chez).
jolt-cf1q.6
The corpus certifier (test/conformance) flagged four cases where jolt's
hand-written :expected matched a real defect rather than Clojure. Fixed in the
jolt-core overlay, corrected the spec :expected, re-certified against JVM Clojure:
- ex-message: returns nil for a non-throwable (dropped the lenient string branch);
still returns the message for ex-info. (jolt-l8e8)
- munge: preserves the argument's type — a symbol munges to a symbol, not a string.
(jolt-hc35)
- print: (print nil) emits "nil", not "" (top-level nil guard; str yields "").
(jolt-pqio)
- bounded-count: uses the counted? fast path (full count), else counts up to n via
seq — was (min n (count coll)), wrong for counted colls. Added an uncounted-coll
spec case. (jolt-2507)
Removed the 4 :bug entries from known-divergences.edn (now certified), regenerated
corpus + profile, re-minted the Chez bootstrap seed (clojure.core changed). Gates:
Janet 155/0, JVM certify clean, both Chez corpus gates 2534 (floors raised),
bootstrap 6/6, fixpoint intact.
The map build already used a transient map, but each bucket was rebuilt with
a persistent (conj (get ret k []) x) per element — an O(log n) trie path
rebuild + alloc each. A coarse grouping (few large buckets) was bound on that
conj, not the map build. Buckets are now native arrays (transient vectors,
O(1) push) frozen once; distinct keys are tracked in a side vector so the
buckets freeze in place with no second map rebuild. A bucket's first element
stays a cheap persistent [x] and only promotes to a transient on the second,
so an all-singletons grouping pays no transient alloc.
coarse (10/100 buckets, 50k): ~313ms -> ~125ms (~2.5x)
2 buckets (50k): ~322ms -> ~129ms (~2.5x)
all-unique (50k): ~949ms -> ~892ms (no regression)
Surfaced a latent bug: canon-key returns nil for a nil key and Janet tables
drop a nil key, so the canon-keyed transient map silently lost a nil-key
entry — group-by/frequencies/assoc!/into{} dropped the whole nil bucket
((group-by identity [nil nil 1]) gave {1 [1]}, not {nil [nil nil], 1 [1]}).
Route nil through a sentinel (tbl-key) at the transient-map keying sites;
persistent!/count/dissoc! work unchanged since the real [nil v] pair is kept
as the stored value, and phm already has its own has-nil slot. The transient
set has the analogous bug (needs phs nil support) — filed separately.
Co-authored-by: Yogthos <yogthos@gmail.com>
into {}, frequencies, group-by, set, into #{} and persistent! all built
their result by folding an immutable assoc/conj per element — each call
rebuilt the O(log32 n) trie path and allocated a fresh wrapper. Add a
one-pass bottom-up HAMT builder (phm-from-pairs) and route the builders
through it, the map/set analog of the pvec bulk build in #153.
phm-from-pairs partitions entries by hash and constructs the bin/array/
collision nodes directly, with the same bin<=16 / array-node>=17 promotion
the incremental path uses — so the trie is byte-identical to one built by
phm-assoc (validated across the size and branching boundaries, including
hash collisions, duplicate keys and the nil key). persistent! map/set and
the set constructor bulk-build; into {} keeps the small-scalar-map-stays-a
-struct rule via bulk-map-from-pairs; frequencies/group-by switch to the
canonical transient form and ride the fast persistent!.
50k A/B: into {} 704->270ms, frequencies 582->160, set 615->241,
into #{} 702->240, group-by 1358->919 (bound on persistent vector conj).
Gate: conformance x3, full suite (4718 >= baseline), new maps/sets bulk
boundary specs.
Co-authored-by: Yogthos <yogthos@gmail.com>
A protocol method reads its fields through the generic guarded keyword lookup
because the method's `this` param is untyped. defrecord now hints `this` with
the record type, the per-form inference seeds ^Record-hinted params (the
:fn branch previously typed all params :any — only the whole-program path
seeded phints), and run-passes feeds the inference the record shapes. So a
hinted param's field reads bare-index instead of going through the :jolt/type
tag guard.
This needed a with-meta fix: (with-meta sym ..) returned a proto'd table, so
symbol? was false and the macro-attached hint broke fn destructuring. Symbols
now carry metadata in-place in their struct (matching how the reader attaches
^hint), keeping symbol? true, as in Clojure.
Modest on dispatch (~3-5%): the field read is a small fraction of a dispatch;
the machinery (record-tag + protocol lookup + wrapper) dominates, which is the
inline-cache target (jolt-ez5h). But it's a correctness fix and lets any
^Record-hinted code — not just methods — drop the field-read guard per-form,
not only under whole-program.
Co-authored-by: Yogthos <yogthos@gmail.com>
Sorted collections were a sorted VECTOR — insert-at = (into (conj (subvec es
0 i) x) (subvec es i)) is O(n) per assoc with a large constant, so building was
O(n^2): 2000 entries took 55.6s.
Replace the rep with a red-black tree (assoc/dissoc/get/contains O(log n)),
ported from the ClojureScript PersistentTreeMap (cljs.core: tree-map-add /
balance-left / balance-right / tree-map-append / balance-*-del). This tier (25)
loads before 30-macros so deftype isn't available; a node is a plain vector
[color k v left right] and cljs's BlackNode/RedNode methods become functions —
the algorithm is unchanged. A sorted-set stores elements as keys with a nil
value; its ops project the key.
The seed read the old :entries vector directly for equality/printing; route
those through a new :entries op that materializes ascending from the tree
(core_types/sorted-entries-arr + main.janet's printer).
2000 sorted-map assocs: 55.6s -> 0.98s (57x); now O(log n) (per-op cost flat
from n=2000 to 10000). Correctness in test/integration/sorted-rbtree-test.janet
(shuffled insert ordering, delete rebalancing, custom comparator, comparator
lookup, subseq, count); sorted specs + full gate green. (key/val on sorted
entries stays a pre-existing gap — entries are pvecs not host tuples; jolt-jk23.)
Co-authored-by: Yogthos <yogthos@gmail.com>
A deftype field tagged ^:unsynchronized-mutable / ^:volatile-mutable is set!-able,
but under direct-link immutable records are shape-rec tuples, so set! errored
("Can't set! field on non-deftype: tuple").
A deftype with any mutable field now opts out of the shape-rec layout and uses
the existing :jolt/deftype table form regardless of :shapes? — set! already
mutates that form and field reads route through the tagged-table path. Such a
type is also not registered as a shape, so the inference never emits a bare-index
read against the table. Immutable deftypes/records keep the fast shape-rec.
deftype extracts per-field mutability from the field metadata and passes it to
make-deftype-ctor, which picks the representation at ctor-build time.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Protocol/interop fixes to run metosin/malli
Bringing up malli (schema validation) surfaced a batch of protocol and host-interop
gaps. m/validate now works across the schema vocabulary (predicates, :map incl.
nested/optional, :vector, :tuple, :enum, :maybe, :and, bounded int/string).
- extend-type and reify now accept MULTIPLE protocols in one form (each bare
symbol switches the current protocol). reify records every protocol it
implements, so instance?/satisfies? recognise all of them.
- Protocol method params support destructuring: reify/extend-type/deftype/
defrecord emit (fn ...) (which desugars patterns) instead of raw fn*.
- instance? of a PROTOCOL works like satisfies? for reify/record instances,
matching short names across qualified/bare protocol references.
- @x reads as the qualified clojure.core/deref, so it still derefs where a ns
excludes and rebinds deref (malli does). Updated reader-test + the reader
spec/grammar (S11, deref rule).
- Java collection interop on jolt collections: .nth/.count/.valAt/.get/.seq/
.containsKey route to the clojure.core equivalent (1-arg and 0-arg paths).
- java.util.HashMap capacity/load-factor constructors + .putAll.
- A class used as a value resolves to its instances' type, so Pattern -> the
regex type (malli keys class-schemas by it).
- Shims for malli's load path: LazilyPersistentVector/createOwning and
PersistentArrayMap/createWithCheck statics.
m/explain not yet working (jolt-fjb1). Full gate green.
* satisfies? recognizes reify, consistent with instance?
A reify's protocol methods are instance-local, so they aren't in the global type
registry that type-satisfies? consults — satisfies? returned false for a reify
even when it implemented the protocol. Check the protocols the reify records on
itself (the same :jolt/protocols list instance? uses), matching short names like
instance? does. Covers single- and multi-protocol reify.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add architecture refactor plan
Synthesizes a six-part architectural review into phased, gate-validated cleanup
work. Targets LLM-maintainability: one home per feature, no god-files, explicit
checked contracts, no copy-paste dispatch. No code changes yet — the plan only.
* Refactor phase 0: dead code + isolated bugs
Pure cleanup ahead of the structural phases (docs/architecture-refactor-plan.md).
No behavior change except the two bug fixes, which are covered by a regression row.
Dead code (all verified zero-reference or overridden):
- core-resolve / core-satisfies? / core-type->str seed stubs + bindings —
resolve and satisfies? are interned by install-stateful-fns! (the seed copies
were shadowed); type->str was an inert SCI stub with no callers.
- find defined twice in 20-coll.clj; the dead copy returned a plain vector
(wrong — the live def at :787 returns a real map-entry) with a comment that
contradicted it.
- mark-hint (passes.clj), phs-to-struct (phm), shape-vals / ns-imports-fn
(types) — unreferenced.
- redundant local pad2 in javatime (module-level one already in scope).
Bugs:
- File.toURL stored :url but every :jolt/url method reads :spec, so a URL from
(.toURL file) returned nil from all its methods. Now stores :spec (+ spec row).
- pl-rest had a no-op (if (plist? r) r r); collapsed to r.
- :map-shapes? was missing from the deps-image cache key — two runs differing
only in map-shapes could reuse each other's image.
Also dropped read-quote's unused pos param. Full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
A record field can carry a type hint — ^Vec3 (a defined record type) or ^:num —
and the inference now resolves it so reading the field back yields that exact type
instead of :any. A Vec3 stored in a Ray field reads out as Vec3, so the vec ops on
field-read values prove their reads (bare-index). This is Stalin's per-slot type
sets, but DECLARED rather than inferred: the exact shape is known up front.
- deftype captures each field's :tag / :num metadata (was stripped) and passes it
to make-deftype-ctor; the ctor registers per-field tags, resolving a record-type
hint to its ctor-key (same-ns) so the inference can look it up directly.
- call-ret-type builds a record's struct type with field types resolved from the
hints, recursing into nested record types (depth-bounded for self/cyclic types).
Measured: a nested-record read loop (:r (:origin ray)) runs 1.3s with ^Vec3 hints
vs 7.1s without — 5.5x. This is the lever the ray tracer needed (vecs flow through
container fields); records without it read back as :any and stay unproven.
A protocol method call compiles to (protocol-dispatch proto method this rest) — a
runtime registry walk (type-tag -> proto -> method) on every call, ~19x a direct
call. When the inference proves the receiver (arg 0) is a known record type, the
call now resolves to a DIRECT method call at compile time, skipping the registry.
- defprotocol registers each method's var-key 'ns/method' -> [proto method] (a
ctx-capturing register-protocol-methods! emitted into the do-block); infer-unit!
feeds it to the inference via a box (like record-shapes).
- the record-ctor return type carries :type (the record tag) so the inference
knows the receiver type; the :else invoke case annotates a protocol call whose
arg0 has a known :type with :devirt-{type,proto,method}.
- emit-invoke resolves the impl via find-protocol-method at emit time and emits a
direct call to the embedded impl fn value. Unknown/polymorphic receivers (no
proven :type) fall back to the dispatch path unchanged.
Measured: removes the dispatch overhead (14.7s -> 9.3s on a 10M-call loop); the
remaining cost is the method body itself (non-inlined, unproven reads) — inlining
the resolved method is the follow-up (jolt-t6r) toward direct-call speed.
Sound under the closed-world assumption direct-linking already makes (the impl is
resolved + embedded at compile time). Adds devirt-test (subprocess: dispatched ==
devirtualized across polymorphic dispatch, unknown-receiver fallback, and
heterogeneous collections). Stalin's compile-call/callee-environment is the model.
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
File API (jolt-hjw): io/file and (File. …) build a tagged :jolt/file value
(instance? File true) with a full method surface (isFile/isDirectory/exists/
getName/getPath/getAbsolutePath/listFiles/toPath/delete/createNewFile/…) backed
by os/ and file/. file-seq is File-aware (leaves are File values). str/slurp/spit
coerce :jolt/file to its path. ClassLoader/getSystemClassLoader + a classloader
stub whose getResource returns nil degrade migratus's classpath lookup to the
filesystem. java.nio.file Path/FileSystem/PathMatcher are shimmed just enough for
script-excluded?'s glob (recursive * / ? matcher).
Three bugs found getting migratus's migration discovery to work:
- (assoc nil k v) returned a raw janet table, not a map, so assoc-in built tables
that count/seq rejected. Now returns an immutable map.
- methods/get-method resolved the multimethod symbol at runtime in the current
ns, so a bare multifn ref in its defining ns saw an empty table once defmethods
lived elsewhere. Now they take the multimethod VALUE and recover the var via a
registry (Clojure semantics).
- defmulti now drops a leading docstring/attr-map (migratus's multimethods carry
docstrings) instead of treating the docstring as the dispatch fn.
Conformance 335/335 x3, clojure-test-suite at baseline.
For the migratus next.jdbc shim (jolt-0z5):
- core.janet: __jdbc-wrap-conn / __jdbc-conn-raw / __jdbc-make-stmt builtins.
A connection is a tagged wrapper over a jdbc.core conn carrying a clj :exec
callback so the host Statement.executeBatch runs SQL without a janet->clj call.
- javatime.janet: tagged-methods for :jolt/jdbc-conn (setAutoCommit/isClosed/
close/getMetaData), :jolt/jdbc-meta (getDatabaseProductName), :jolt/jdbc-stmt
(addBatch/executeBatch/close); java.sql.Timestamp ctor -> millis.
- evaluator.janet: instance? case for Connection/java.sql.Connection so
migratus's do-commands runs SQL through its Connection branch.
Two defn/defmacro fixes found loading migratus.core (both rooted in the reader
representing ^{:map} name metadata as a with-meta form, jolt-8w2):
- defmacro special form: unwrap a with-meta name (mirrors def), and handle the
arity-clause form (defmacro name ([params] body...)) like fn/defn — a params
vector reads as a tuple, an arity clause as a list (array).
- defn overlay: pass the bare (unwrapped) name to fn while def keeps the meta.
Conformance 335/335 x3 modes.
A typo'd symbol used to auto-intern an unbound var and die later as
'Cannot call nil as a function' with no hint which symbol. Now:
$ jolt -e '(undefined-fn 1)'
Error: Unable to resolve symbol: undefined-fn in this context
The analyzer's :unresolved fallthrough now punts to the interpreter
(whose resolver raises the message above when the form runs) instead of
emitting a var-ref that interned the var. A punt rather than a hard
throw because runtime-interning forms (defmulti's setup) legitimately
reference the var they're about to create from a nested do.
Pulling that thread surfaced three real bugs the leniency was masking:
- h-resolve-global resolved unqualified symbols against ctx-current-ns,
which during analysis is jolt.analyzer — so user-ns vars NEVER
resolved through it; the lenient arm happened to emit the right ns.
Now resolves against the compile ns like the qualified branch.
- Top-level (do ...) wasn't split: Clojure compiles and EVALS each
child in sequence so earlier children's runtime effects (defmulti's
intern) are visible while later children compile. eval-toplevel now
splits.
- The stdlib itself had forward references the auto-intern hid:
10-seq's transducers used vreset!/vswap! from 20-coll (moved to
10-seq); in 20-coll qualified-ident?/realized?/list*/underive
referenced defs declared later in the file (reordered); sorted? and
partition-all are genuinely later-tier and got (declare ...).
Test rows updated where they encoded the old leniency: ir-passes'
dead-branch row (unresolved in a dead branch is an error, as in
Clojure), compile-mode's ctx-isolation row (other ctx now errors
instead of reading nil), cli rows assert the new message. Gate green,
conformance 335/335 x3, suite 4718 steady, bench within noise.
Before: (+ 1 "a") printed 'could not find method :+ for 1 or :r+ for "a"'
followed by three janet frames pointing at jolt internals. After:
Error: Cannot add 1 and "a" — + expects numbers
at app.deep/level3
Round 1 — compiled fns carry their Clojure identity:
- The analyzer's recur target (which doubles as the compiled janet fn's
name) is now ns/fn-name (_r$app.deep/level3--N), so janet stack traces
name the user's fns; defn passes the self-name through to fn.
- eval-toplevel re-raises with propagate instead of protect+error — the
failing fiber's stack was being discarded, which is why every trace began
at eval-toplevel.
- require/maybe-require-ns route loaded namespaces through the loader's
compile-or-interpret eval-toplevel via a ctx hook (the evaluator can't
import the loader). Previously REQUIRED namespaces always ran interpreted:
slower, and their fns were anonymous in traces.
Round 2 — report-error presents for users (rephrase-inspired):
- The full trace text is stashed at the innermost eval-toplevel boundary
(janet's debug/stacktrace walks the fiber propagation chain; debug/stack
cannot), then filtered: _r$ frames demangled to ns/fn-name, jolt-internal
and [eval] frames dropped. JOLT_DEBUG=1 restores the raw janet trace.
- Message rewrites: janet arithmetic dispatch -> 'Cannot add X and Y — +
expects numbers'; compiled arity -> Clojure's 'Wrong number of args (N)
passed to: ns/fn'; nil-call gets an undefined-symbol hint (round 3 will
fix resolution properly).
6 cli-test rows assert the exact user-visible output. Gate green, suite
4718 steady, bench within noise.
Clojure's defprotocol takes an optional docstring and leading keyword
options (:extend-via-metadata true) before the signatures; jolt's macro
fed the option keyword to (first sig). honeysql declares its InlineValue
protocol exactly that way — with the fix, all four honeysql namespaces
load unmodified from git and the formatter produces correct sqlvecs for
selects/inserts/updates/deletes/joins/:inline. Listed in libraries.md.
(^bytes [b])-style return hints reach fn as a (with-meta [b] {:tag ...})
form; unhint sheds the wrapper through the rebuild path so the clause
representation never changes. The host-interop hint rows exercise it.
The interop surface ring.util.codec needs (registered through the javatime
shim registries): URLEncoder/URLDecoder (www-form-urlencoded in pure janet),
Charset/forName, Base64 encoder/decoder, Integer/valueOf with radix +
parseInt, StringTokenizer, clojure.lang.MapEntry (a 2-tuple), a String ctor
from bytes, .getBytes on the String surface, and a java.lang.Number method
surface (byteValue and friends).
Protocol fixes: extend-protocol on java.util.Map/Set/List now dispatches
(maps — phm/struct/sorted/records — never produced host tags and fell to
Object), lazy seqs gained their ISeq tags, and a nil extension arm works
(group-by-head and extend-type both choked on the nil head). reduce
dispatches to a reified clojure.lang.IReduceInit's own reduce method, which
is how ring-codec tokenizes.
jolt-deps learns :deps/root (tools.deps monorepo subdirectory checkouts —
ring-core lives inside ring-clojure/ring). spork/http, when jpm-installed,
reaches the jolt layer as janet.spork.http/* through the janet.* bridge
(soft: nothing requires it unless used).
The protocol fixes alone let 29 more clojure-test-suite assertions execute:
5319 -> 5348 run, 4706 -> 4715 pass.
The first per-type migration print-method unlocked: uuid, regex, transient,
and channel rendering move from host pr-render branches to io-tier
defmethods (exact same output). The renderer's tagged fallthrough now
dispatches ANY remaining :jolt/* value through the print-method hook before
the raw pairs view — so every tagged type is user-overridable, atoms
included ((defmethod print-method :jolt/atom ...) fires nested), and future
per-type migrations are pure overlay additions.
Hot types (numbers, strings, symbols, collections) stay native, and inst/
namespace/var stay host for now — their formatters (rfc3339, display names)
live there anyway. A transient's :kind is read with jolt.host/ref-get: get
on a transient is the dispatched collection lookup (same trap as sorted
colls).
Before the hook is wired (init-time error messages) tagged values fall
through to the pairs view — bootstrap rendering only.
print-method/print-dup are now multimethods in the io tier with Clojure's
exact dispatch ((:type meta) keyword, else type — core.clj 3693). On jolt the
dispatch value for a record is its quoted full-name symbol, since class names
aren't values here.
Records used to pr-str as the raw janet table; the renderer's record branch
now prints Clojure's #ns.Type{:k v} syntax, and first consults a callback the
api wires up after the overlay loads — so a user defmethod on a record type
fires everywhere: top level, nested in collections, through pr/prn/pr-str.
Builtin overrides (a :number method) fire only on direct print-method calls;
pr keeps the native fast path (documented divergence).
java.io.Writer arrives as a shim beside the StringReader/StringBuilder ones:
a :jolt/writer tagged value with write/append/flush/toString, a StringWriter
ctor, and a sink variant the renderer callback uses.
Two latent host bugs fixed on the way: the interpreted syntax-quote splice
blew up on ~@nil (an interpreted macro's empty & rest binds nil — first tier
user of defmulti found it; d-realize now treats nil as the empty seq), and
(print-method x nil) now throws like the JVM instead of returning nil.
10 spec rows; bench dead even (sandwich run); greeter green on a fresh
binary.
Batch 2 of the post-shrink sweep, all pure compositions or documented stubs:
enumeration-seq/iterator-seq (seq), promise/deliver (an atom — deref of an
undelivered promise stays nil, single-threaded host), bean, uri?,
special-symbol? (an evaluated set of quoted symbols — a QUOTED set literal
stays an unevaluated reader form on jolt, which the first version tripped
over), print-method/print-dup (inert until jolt-g1r), and the whole proxy
surface (mappings/call-with-super/init/update pass through, the constructive
half throws). The seed loses 16 defns and bindings; nothing kept.
