* Add benchmark suite for alloc/dispatch/collection workloads (jolt-1r86)
The ray tracer is float-compute-bound (devirt, alloc removal, type-proving all
measured flat on it), so it can't validate the optimization passes. Add a small
cross-language suite (AWFY + CLBG style, portable Clojure) isolating the axes it
misses:
binary-trees allocation / GC pressure (escaping short-lived records)
dispatch megamorphic protocol dispatch (~1M dispatches/s; WP can't devirt)
collections persistent map/vector churn
bench/run.sh runs them; bench/README.md maps each to the pass it exercises.
collections immediately surfaced jolt-684u: the persistent hash map is O(n) per
assoc (flat copy-on-write bucket array, not a HAMT) — n=4000 assocs take 50s.
Invisible to the ray tracer (no maps).
* Persistent hash map: HAMT instead of O(n) copy-on-write (jolt-684u)
The map was a flat bucket array whose assoc copied the whole array every insert
(O(n)/assoc, O(n^2) to build). Compounding it, small maps are Janet structs that
only promoted to phm for collection keys — never for size — so a scalar-key map
stayed an O(n)-copy struct forever. Building a 4000-entry map took ~50s.
Two fixes, following ClojureScript's design:
- phm.janet is now a HAMT (hash array mapped trie): BitmapIndexedNode /
ArrayNode / HashCollisionNode, 32-way, 5 hash bits per level, structural
sharing — assoc/dissoc/get are O(log32 n). Translated from cljs.core, adapted
to Janet's 32-bit bit-ops (the hash is carried unsigned, the level index is
extracted with arithmetic, and bits are tested with band against 1<<i since
brushift rejects negative bitmaps). The public phm-* API and the value shape
(:jolt/type :jolt/phm, :cnt) are unchanged; transients are a separate rep and
untouched.
- core_coll promotes a struct map to a phm past 8 entries (not only for
collection keys), mirroring cljs PersistentArrayMap -> PersistentHashMap, so
incremental building isn't O(n^2).
20000 raw assocs: 7.1s -> 0.105s. The collections benchmark: 16.7s -> 0.2s.
Correctness covered by test/unit/phm-hamt-test.janet (oracle vs a Janet table,
nil keys, dissoc, a real hash-collision pair, and a sub-linear-assoc guard);
full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
The ray tracer is float-compute-bound (devirt, alloc removal, type-proving all
measured flat on it), so it can't validate the optimization passes. Add a small
cross-language suite (AWFY + CLBG style, portable Clojure) isolating the axes it
misses:
binary-trees allocation / GC pressure (escaping short-lived records)
dispatch megamorphic protocol dispatch (~1M dispatches/s; WP can't devirt)
collections persistent map/vector churn
bench/run.sh runs them; bench/README.md maps each to the pass it exercises.
collections immediately surfaced jolt-684u: the persistent hash map is O(n) per
assoc (flat copy-on-write bucket array, not a HAMT) — n=4000 assocs take 50s.
Invisible to the ray tracer (no maps).
Co-authored-by: Yogthos <yogthos@gmail.com>
A ^Record param hint was applied only at the final re-emit (reinfer-def), not
during the inter-procedural fixpoint. So a hinted param with no callers stayed
:any while inference ran, and a field read off it (e.g. (:origin ^Ray r)) never
told a non-inlined callee that its arg is a Vec3 — the callee's params stayed
unproven and its field reads kept the dynamic guard.
Seed declared hints as a param-type floor in the fixpoint: phint-seed (passes/
types) resolves an arity's :phints to positional record types via the
record-shapes registry, and infer-unit! initializes each fn's fresh param slots
from them instead of nil. A fixed declared type can't poison the least-fixpoint
the way an early-iteration :any would, and a hinted param now propagates its
(and its field reads') types to its callees during inference.
Scope: this closes the hinted-propagation gap. It does NOT help the ray tracer,
which uses zero ^-hinted params (only hinted fields) — its remaining type gap is
unhinted record-param inference on recursive/non-inlined hot fns, and per the
jolt-15jq A/B it's allocation-bound regardless (jolt-8flj). Tracked on the bead.
Co-authored-by: Yogthos <yogthos@gmail.com>
Dependency resolution now lives in the `jolt` CLI itself instead of a separate
jolt-deps executable. `jolt` resolves a deps.edn into JOLT_PATH/JOLT_APP_PATHS
in-process and dispatches the deps subcommands:
jolt -M:alias [args] run the alias :main-opts
jolt -A:alias CMD run CMD with the alias paths
jolt run FILE resolve, then run FILE
jolt path | tasks | task NAME
A deps.edn in the working dir is auto-resolved for the runnable commands
(repl/-m/-e/nrepl-server/FILE), so e.g. `jolt -M:nrepl` (or plain
`jolt nrepl-server`) starts an nREPL with the project and its deps loaded.
The runtime core stays deps-agnostic — it only reads JOLT_PATH. The resolver
(deps.janet) is reached only from the CLI entry and loads jpm lazily, so a run
with no deps.edn never touches it and an app baked from its own jolt/api entry
never links it. resolve-deps-argv only resolves on an explicit deps command or
when a deps.edn is present; help/version never do.
jolt-deps stays as a thin deprecation shim that forwards to `jolt`, so existing
scripts keep working. Docs (README, CLAUDE.md, building-and-deps, tools-deps)
and the help text updated.
Co-authored-by: Yogthos <yogthos@gmail.com>
scalar-replace already folds non-escaping const-key map literals
((:k {:k a ..}) -> a, and drops a let-bound map that doesn't escape).
Extend the same fold to record constructors: a (->Rec a b c) is a
positional struct whose declared field order lives in the record-shapes
registry, so a field read on a non-escaping ctor folds to the matching
positional arg and the allocation disappears.
Direct form (:field (->Rec ..)) and the let-bound form both handled,
threaded through run-passes via a per-unit shape registry (new
jolt.host/record-shapes accessor). Soundness: ctor args must be pure
(duplicated/discarded like map vals), arg count must equal the field
count, and only declared-field reads fold — a record answers the virtual
:jolt/deftype key with its type tag and any other key with nil, neither
of which is a positional arg, so those keep the allocation. pure? now
treats a record ctor of pure args as pure, so nested records (a Ray
holding a Vec3) fold bottom-up.
Allocation-bound microbench (non-escaping record built + field-read in a
hot loop): 69.6s -> 2.4s, landing on the no-record arithmetic baseline.
The ray tracer is unchanged — its vec3 results escape (returned/stored
each op), so they genuinely allocate; that's a separate problem.
Co-authored-by: Yogthos <yogthos@gmail.com>
cap truncates a deep type's field VALUES to :any so the inter-procedural
fixpoint stays finite, but it rebuilt the struct via mk-struct and dropped the
record :type tag along the way. The tag is identity — independent of field
depth — so a record stored in a deep container (a Sphere in a world vector, a
material on a hit) degraded to a plain struct, and devirtualization (jolt-41m)
and record? folding silently stopped firing on it.
Preserve :type alongside :shape when capping. Verified: a protocol call on a
record read out of a vector now devirtualizes (the call node gets :devirt-type,
which needs the receiver's record type). Sound — the tag stays accurate; only
field values below the depth cap are truncated.
No measurable wall-clock change on its own (jolt's protocol dispatch is already
cheap), but it restores the record fast path / devirt / record?-folding on
records-in-containers, and unblocks downstream work that keys off record types.
Co-authored-by: Yogthos <yogthos@gmail.com>
A bundle is closed-world — everything it needs is inlined and nothing is
required afterward — so a user defn unreachable from the entry's reference
graph can be dropped. The bundler now computes reachability from main-ns/-main
plus every non-prunable form and drops dead defn/defn- by exact source span
(formatting and reader macros in the surviving code are untouched).
Conservative and sound: only plain defn/defn- are prunable; a defn is kept if
its bare or ns-qualified name appears in any kept form, the closure runs to a
fixpoint, and any use of resolve/ns-resolve/requiring-resolve/find-var/intern/
eval/load-string disables pruning entirely. A parse failure on any file also
falls back to verbatim bundling, so the command stays as robust as a plain
concatenation. defmethod/defrecord/extend bodies are non-prunable and scanned,
so a fn reached only via dynamic dispatch stays live.
New reader/parse-all-spans returns [form start end] byte offsets so the drop
is a verbatim slice, not a re-print.
30-fn library used by a 3-fn entry: bundle 1114 -> 437 bytes (61% smaller, 27
dead fns dropped), output byte-identical.
Co-authored-by: Yogthos <yogthos@gmail.com>
When the collection-type inference proves an argument's type, number?/
string?/keyword?/record?/nil?/some? fold to a compile-time boolean. A
const-fold now runs after inference so a folded predicate propagates and
collapses any if it gates to the taken branch.
Sound by construction: only a provable answer folds, and only when the
argument is side-effect-free (a const or local) so dropping its evaluation
is a no-op. Unknown types (:any/:truthy) and impure args keep the call.
vector?/set?/map? are left out — the :vec tag conflates a real vector with
a range/seq, so vector? could be wrong.
50M-iter loop, same shape isolated with a carry-only control: number? call+
branch 5080ms, predicate folded 1365ms — matching the 1417ms control floor,
so the 3.7x is entirely the eliminated call+branch.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Reader: #() params survive syntax-quote (auto-gensym names)
#(...) named its synthesized params with bare gensyms, so a #() written inside a
syntax-quote had its params qualified to the current ns by sq-symbol — and a
qualified symbol isn't a valid fn param. hiccup's compiler emits
`(let [sb# ..] (iterate! #(.append sb# %) ..)), which broke with "Unable to
resolve symbol: ns/_NNNN".
Name the params with a trailing # (auto-gensym suffix, like Clojure's p1__N#) so
syntax-quote maps them consistently and leaves them unqualified. Harmless outside
a backtick (just a regular symbol name).
* interop: String/valueOf static + String is a CharSequence
Two interop gaps surfaced bringing up hiccup and malli:
- String/valueOf(Object): hiccup's compiler stringifies attribute values with
(String/valueOf (or arg "")). Added the static — "null" for nil, else core-str.
- (instance? CharSequence s) returned false for a string; String implements
CharSequence, and malli's :re validator gates on it before matching, so :re
schemas always failed. instance-check now answers true for strings.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Under JOLT_OPTIMIZE a -m program run inferred + specialized EVERY loaded
namespace, including every transitive dependency. On a dep-heavy app that's
prohibitive: malli-app cold-started in ~2m10s (hundreds of dep namespaces, each
run through the per-form inline + inference passes).
The closed world a whole-program pass reasons over is the APP, not its
libraries. jolt-deps now passes the project's own source roots (its deps.edn
:paths) to the runtime as JOLT_APP_PATHS. A namespace loaded from an app root
gets full optimization (and joins the one whole-program fixpoint); a dependency
namespace compiles at default cost — :inline? off for its load, so the per-form
optimize passes don't run over library code — staying direct-linked but
generically typed (the open-world default). With no app roots declared (a bare
program run, or jolt without jolt-deps) everything counts as app, so behavior is
unchanged.
malli-app JOLT_OPTIMIZE cold start: 2m10s -> 4.5s. Compute-heavy programs whose
hot code is their own namespaces (the typed ray tracer) are unaffected — their
code is app code and still fully optimized (9s/frame render). Applied at runtime
in main for the same baked-at-build-time reason as JOLT_PATH; added to the
ctx-image cache key. Help text corrected: optimization is opt-in, not default.
Co-authored-by: Yogthos <yogthos@gmail.com>
A deftype field tagged ^:unsynchronized-mutable / ^:volatile-mutable is set!-able,
but under direct-link immutable records are shape-rec tuples, so set! errored
("Can't set! field on non-deftype: tuple").
A deftype with any mutable field now opts out of the shape-rec layout and uses
the existing :jolt/deftype table form regardless of :shapes? — set! already
mutates that form and field reads route through the tagged-table path. Such a
type is also not registered as a shape, so the inference never emits a bare-index
read against the table. Immutable deftypes/records keep the fast shape-rec.
deftype extracts per-field mutability from the field metadata and passes it to
make-deftype-ctor, which picks the representation at ctor-build time.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Don't direct-link a var redefined earlier in the same unit (jolt-wf4)
defrecord/deftype expands to (do (def R (make-deftype-ctor ...)) (def ->R R) ...),
so the ->R alias references R within one compiled unit. Under direct-link a var
ref embeds the cell's root as a compile-time constant, but on a redefine R's old
root is still in place when that unit compiles — the (def R new) sibling hasn't
run yet — so ->R sealed to the stale pre-redef ctor. (defrecord R [x a])
(defrecord R [a x]) (:a (->R 10 20)) read the old [x a] layout and returned 20.
Track the vars a unit (re)defines and force their later in-unit references to the
live indirect deref. The cell is registered only after its own init is emitted,
so a recursive self-reference inside the init still direct-links (it runs after
the def completes); only sibling references after the def go indirect.
* Emit Janet's `in` as a value so a user local can't shadow it (jolt-fjb1)
The back end emits `in` to deref var cells ((in cell :root)) and index
shape-recs. It emitted the bare symbol, so a user local named `in` shadowed
Janet's builtin in the surrounding scope and the generated cell-deref called the
user's value as a function — "<table> called with 2 arguments, possibly expected
1". malli's explainer binds [value in acc], so m/explain hit this on every
schema (m/validate was unaffected — its path doesn't bind `in`).
Embed `in`'s function value at the emit sites (as jolt-call/core-get already
are); a value in head position can't be shadowed. Fixes m/explain on malli
(loaded with JOLT_FEATURES=clj so its .cljc reader-conditionals resolve).
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
eval-form treated only a reader LIST (a Janet array) as a call; a runtime-built
list — a plist or lazy-seq from cons/concat/list or ~@ (list?/seq? true but
array? false) — fell through to self-eval. So (eval (cons '+ '(1 2))) returned
the list as data instead of 3, and a macro whose output contained such a subform
left it unevaluated. Add a plist?/lazy-seq? branch that coerces to the element
array via d-realize and dispatches through eval-list; an empty list self-evals.
The analyzer already punts these forms to the interpreter (analyze's :else ->
uncompilable -> interpreter fallback), so this one interpreter branch fixes the
correctness bug across the eval and macro-expansion paths; compiling them
directly (vs punting) would be a separate perf change. Verified: conformance
355/355, syntax-quote ~@ splice, list values unchanged.
A deftype with (Object (toString [_] s)) had its toString ignored: the generic
object-methods "toString" fired in dispatch-member before the record's own
method (the record isn't a tagged shim, so that guard passed), and str rendered
the #Type{...} data repr instead of routing through toString.
- dispatch-member: a record's own method (instance/reified/protocol) now wins
over the generic object-methods table — so .toString/.equals/.hashCode on a
record use the record's definitions; plain records still reach object-methods.
- str: add a late-bound record-tostring-cb (wired per-ctx by
install-print-method-cb!, mirroring print-method-cb) that str-render-one
consults for records — a deftype with a custom toString renders via it, plain
records keep the data repr. pr-str is unchanged.
Needed by hiccup's RawString. Adds deftype-tostring-spec (.toString + str +
concatenation + a regression guard that record-less-toString keeps its repr).
walk only handled vector?/map? and fell through :else for everything else, so
postwalk over a quoted list (a plist) never touched its elements —
postwalk-replace with symbol keys silently no-op'd, which broke
clojure.template/apply-template (found during reitit work). Add list? (rebuild as
a list) and seq? (map over it) branches after the vector/map ones so concrete
collections stay authoritative. Adds walk-spec covering list/seq walking plus a
vector/keywordize regression guard and the apply-template trigger.
The 813-line host_interop.janet (java.lang statics, java.time, the 458-line
java.io/util/net/sql/text install-io!, and collection interop all in one file)
becomes a 19-line aggregator over src/jolt/interop/:
- java_base.janet — java.lang statics (Math/Thread/System/Long) + the java.time
shim + the shared chr/pad2/formatter coercion helpers
- host_io.janet — java.io/util/net/sql/text shims; imports java_base for the
shared helpers it reuses (it constructs java.time values and
formats dates)
- collections.janet— the late-bound .iterator/.nth/.count/.seq interop hooks
host_interop.janet just loads the three and runs install!/install-io!/
install-collections! in order. chr/pad2/formatter become public (they cross the
java_base->host_io boundary now). The registry machinery (class-statics/
tagged-methods/register-*!) stays in the evaluator, which loads first — moving it
out touches the hot dot-dispatch and is left as a separate step.
Pure move, no behavior change. Also fixes a cache footgun the subdir exposed:
source-fingerprint walked src/jolt/ with a non-recursive (os/dir), so an edit to
an interop/ file would not have busted the ctx image cache (stale-image bug) —
made it recurse, keyed by repo-relative path.
Full gate green: host-interop-spec (130+ rows across every JDK area), ctx-image
cold/warm, conformance x3, suite >=4695/88, fixpoint.
init-cached (core image) and the deps-image (main.janet) each hand-wrote the same
fork->slurp->validate-ctx?->rewire load and the snapshot->tmp->atomic-rename save.
Extract load-ctx-image / save-ctx-image (in api.janet, beside snapshot/fork): the
two callers now differ only in the validity predicate they pass — none for the
core image (its source fingerprint is already in the path), a deps-manifest mtime
check for the deps image. The per-process print-method-cb rewiring an image
restore must replay is threaded as a callback so the helpers don't depend on core.
Kept in api.janet rather than a new ctx_image.janet module: Janet's `use` doesn't
transitively re-export, and snapshot/fork already live in api and are consumed via
(use ./api) by main and the test harnesses — a separate module would force every
caller to import it directly. (load-image/save-image collide with Janet builtins,
hence the ctx- prefix.)
Full gate green; ctx-image-test cold/warm + deps tests pass.
core-count/core-seq/core-conj each walked a chain of (and (table? x) (= :jolt/X
(get x :jolt/type))) predicates — re-fetching the type tag per predicate and, for
conj, a 14-deep nested if. Replace with a single (case (get coll :jolt/type) ...)
per op: the type is read once and the arm calls the concrete op directly. Host
values (tuple/array/nil) and tuple-based shape-recs carry no :jolt/type and stay
in a per-op fallback cond (shape-rec? kept before tuple? — shape-recs ARE tuples).
Factor the two map-conj paths (phm vs struct) into one conj-into-map parameterized
by the assoc fn.
Behavior-preserving: the :jolt/type tags are disjoint, so the case is an exact
re-expression of the predicate chain; the fallbacks reproduce the originals
(incl. count erroring on a raw host table, which only seq ever handled).
This is the perf-neutral realization of the planned collection vtable. A shared
:jolt/type->{ops} table was tried first and REGRESSED core-bench ~2-4% (table
lookup + indirect call on the hot path); the inline case instead runs ~1.7%
FASTER than main (4864ms vs ~4950ms baseline) since one type-get + a jump beats
the sequential predicate chain. Full gate green (suite >=4695/88, fixpoint).
analyze-try assoc'd :catch-sym/:catch-body/:finally nil-when-absent, so a try
with no catch (or no finally) carried a nil-valued key — which makes the node a
phm in jolt's map representation and forces the back end to densify it
(norm-node) before reading :op. That's the map-nil-representation trap Phase 2
already cleaned up for def/fn/arity nodes. Add those keys only when the clause is
present, matching the arity :rest discipline; a try node stays a fast struct.
Behavior-invisible: emit-try reads each key with a nil-safe (node :k) and gates
on it, so an absent key and a present-nil key are indistinguishable to every
consumer. Adds ir-try-shape-test asserting the node shape across all four
try/catch/finally combinations plus end-to-end eval.
Note on scope: the plan's "delete the defensive norm-node calls" is NOT done — it
can't be. {:op :const :val nil} (e.g. (def x nil)) and nil map keys are
inherently phm, so the emit-dispatch norm-node guards a real case, not a
present-or-absent artifact. This PR removes a source of gratuitous phm nodes
rather than the densification itself. Full gate green.
Six bottom-up IR rewrites (const-fold, inline-node, subst, flatten-lets,
subst-lookup, scalar-replace) each hand-listed every op's child positions —
~250 lines of identical "recurse children, rebuild" arms that had to be kept in
sync whenever an op was added. Extract one map-ir-children into ir.clj that
knows each op's child layout; each walk keeps only its genuine specials
(const-fold's invoke/if, inline-node's invoke, subst's local/let alpha-rename,
scalar-replace's invoke/let folds) and delegates the rest.
The combinator is total over the op set, so the walks are now total too: a
couple soundly gain coverage they previously skipped (const-fold now folds
inside :try; subst-lookup now recurses :def inits, which fixes a latent dangling
ref where a dropped const-key-map binding was referenced inside a def). These
are sound — all six are result-preserving optimizations — and 3-mode conformance
+ fixpoint confirm identical program behavior.
map-ir-children is shape-preserving for :try (recurses :catch-body/:finally only
when present, never assoc's nil) so it can't turn a struct node into a phm.
Written with cond/get only, matching the passes' tier, so no new load-order dep.
Predicates (body-closed?/pure?/local-escapes?), the type-threading infer, and the
Janet backend emit stay as-is: their conservative :else defaults / [type node]
threading / host language don't fit a node-rebuilding combinator.
Adds ir-passes-test coverage for folding reaching fn/loop/try bodies. Full gate
green (conformance x3, suite >=4695/88, fixpoint stage1==2==3, inline-sra + devirt).
eval-dot copy-pasted its entire dispatch chain across the (. obj method args...)
and (. obj member) forms — string/number/object/tagged-shim lookup duplicated,
hand-synced on every interop change. Extract one dispatch-member that takes the
evaluated args plus a has-args flag. The shared head (string/number/object/
tagged) is single-sourced; the genuinely divergent tails (call form: record →
native field → coll-interop(args); bare form: zero-arg coll-interop → field /
zero-arg method) stay branched on has-args. The guards that differed between the
arms (object-methods checks table? only; tagged dispatch checks table-or-struct;
bare-form tagged dispatch requires the member present) are preserved verbatim,
keyed off has-args, so behavior is identical.
Adds a "dot dispatch arms" spec locking the divergent cases: zero-arg vs
with-arg coll-interop, record/deftype zero-arg vs with-args methods, -field
access. Full gate green.
5d: document the seed↔overlay boundary and add a drift check. core fns split
across a Janet seed (core-X, registered in core-bindings) and a Clojure overlay;
five names (char?/sorted?/sorted-map?/sorted-set?/transduce) carry a defn in
both, with the overlay copy authoritative and the seed copy internal-only. The
into-vs-transduce home asymmetry was undocumented. Adds docs/seed-overlay-registry.md,
SEED-TWIN: comments at the five seed sites, and a build-time drift check
(test/unit/seed-overlay-registry-test.janet) that recomputes the twin set from
source and fails if it diverges or a twin leaks into core-bindings.
5e: rep↔API pointer comments in pv/plist/phm/phs/lazyseq (representation lives
here; Clojure-facing ops dispatch in core_coll/core_types) and back-pointers in
core_coll. No behavior change — comments, docs, one source-analysis test.
Full gate green (suite ≥4695 pass / ≥88 clean files), drift check passes.
main.janet held ~45 lines of env-knob policy (open-mode / direct-link / optimize
/ shapes / whole-program gates) that couldn't be unit-tested without the CLI, and
two disk-image caches (api/init-cached, main/deps-image) each hand-built a
POSITIONAL "%q|%q|..." key that silently misaligned if a ctx-shaping knob was
added in only one place.
config.janet now owns:
ctx-shaping-env-vars the canonical list of env vars that shape the built ctx
ctx-cache-key a labeled key (name=value) over a prefix + every shaping
var, so adding a knob updates BOTH cache keys at once and
can't positionally alias two different builds
resolve-run-mode [open-mode? main-entry?] -> the ctx env knob map
main shrinks to: compute open-mode?/main-entry? from argv, call resolve-run-mode,
install the knobs. Both deps-image-path (main) and image-cache-path (api) build
their keys via ctx-cache-key. New test/unit/config-test.janet locks in the
run-mode cases and asserts every ctx-shaping env var participates in the key.
Scope: this is 4a + the 4b cache-key footgun fix. The optional 4b cleanup
(folding the load/save image dance + aot marshal helpers into one ctx_image
module) is left for a follow-up — it's lower value and higher blast radius.
No behavior change (cache keys now key on a superset of env vars, so at worst a
one-time cold rebuild). Gate green: conformance 355x3, clojure-test-suite 4718
pass (>= 4695 baseline), config-test, full jpm test exit 0.
Two small structural dedups from the plan's Phase 3c.
reader.janet: read-list/read-vector/read-set each had a copy of the same
read-loop (skip whitespace, stop at close char, drop #_ discards, splice #?@).
The skip/splice logic had drifted between them once. Hoisted into one
read-delimited [s pos close err] -> [items end]; the three readers now just wrap
its result. read-map keeps its own loop (its key/value pairing needs a different
value-slot scan).
phm.janet: phm-bucket-find/contains?/assoc/dissoc and phm-get each open-coded the
same stride-2 key scan. Extracted bucket-index-of [bucket k] -> index|nil; all
five now share it.
No behavior change. Gate green: conformance 355x3, clojure-test-suite 4718 pass
(>= 4695 baseline), full jpm test exit 0.
types.janet held five concerns under one generic name. Split into sibling
modules along its existing section boundaries:
types_symbols characters + symbol helpers
types_var Var
types_ns Namespace
types_ctx Context (+ inst/uuid values)
types_protocols protocol/type registry + shape-records
types.janet is now a pure aggregator: it loads the clusters in dependency order
and re-exports their defs (import :prefix "" :export true), so every consumer
keeps its single (use ./types) unchanged.
Order-preserving (statically verified: zero backward references, zero
cross-cluster private helpers — the cleanest of the seed splits). No behavior
change.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
evaluator.janet was a 2597-line file with a 680-line eval-list. Split into
cluster modules behind a re-export aggregator (same pattern as core):
eval_base forward vars, syntax-quote, ns-loading, registries, jolt-invoke
eval_resolve symbol/var resolution, params, destructuring, class lookup
eval_runtime protocols, multimethods, deftype/reify, install-stateful-fns!
eval_special the special forms (eval-list dispatch)
evaluator.janet stays the module every consumer imports: it loads the clusters
in dependency order and re-exports their defs (import :prefix "" :export true),
so the five (use ./evaluator) consumers are unchanged. It still owns the
eval-form entry that ties resolution + special forms + map/coll evaluation.
In eval_special, the giant eval-list match is exploded: each multi-line arm is
now a named (defn eval-<form> [ctx bindings form] ...) — eval-def, eval-fn*,
eval-let*, eval-loop*, eval-try, eval-set!, eval-dot, etc. — and eval-list is a
thin dispatch table over them. "where is try handled" is now `grep eval-try`.
Order-preserving (statically verified: no symbol used before its cluster loads;
zero backward refs). 27 helpers shared across clusters are now public so `use`
shares them. The two near-duplicate .method dot blocks are NOT merged here — that
is a behavior-sensitive dedup tracked separately (jolt-eos3); this PR is pure
moves + the mechanical eval-list explosion, no behavior change.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
core.janet was a 3013-line grab-bag. Split into six order-preserving cluster
modules:
core_types vector helpers, predicates, math, comparison, equality
core_coll collections, transducers, seqs, HOFs, constructors
core_print string + pr-str/str rendering
core_io I/O, files, JDBC, compare, type
core_refs arrays, bit ops, coercions, hash, atoms/refs
core_extra additional clojure.core fns, transients, hashing
core.janet stays the module everyone imports: it loads the clusters in
dependency order and re-exports each one's defs (import :prefix "" :export true),
so every consumer keeps its single (use ./core) with no change. The aggregator
also owns core-bindings and init-core!, which reference fns from every cluster.
The split preserves definition order exactly (verified: no symbol is used in a
cluster that loads before its definition), so seed load-order semantics are
unchanged. Three private helpers used across clusters (map-entries-of,
map-assoc1, str-render-one) are now public so `use` shares them; the five
forward vars (canon-key/jolt-equal?/pr-render/core-compare/print-method-cb) each
stay within their owning cluster.
No behavior change. Gate green: conformance 355x3, clojure-test-suite 4718 pass
(>= 4695 baseline), full jpm test exit 0.
passes.clj was a 1486-line grab-bag mixing three weakly-coupled concerns. Split
along the clusters the review mapped (only run-passes + the dirty flag were
shared):
jolt.passes.fold const-fold + the shared scalar-const? predicate (base)
jolt.passes.inline inline + flatten-lets + scalar-replace
jolt.passes.types collection-type inference + success checker + driver API
jolt.passes façade: run-passes + :refer re-exports of the driver fns
the back end looks up by name
scalar-const? was used by both the inline pass and the inference walk, so it
moves to fold (the base layer) and both refer it. The check-mode state stays
private to jolt.passes.types behind a new run-inference fn; run-passes calls it.
build-compiler! loads the three in dependency order before the façade, mirroring
the existing jolt.ir -> jolt.analyzer bootstrap. No behavior change. Also fixed
the stale ns docstring that listed four passes and omitted the type system.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
Completes the phm.janet decomposition (jolt-bvek): after lazyseq left, the set
follows. phm.janet is now purely the PersistentHashMap; phs.janet is the thin
set layer over it (members are keys -> true), and (use ./phm) for the builders.
Importers using set?/phs-* via (use ./phm) add (use ./phs); backend's emitted
set literal head changes phm/make-phs -> phs/make-phs. Behaviour unchanged
(sets verified interpreted + compiled; full gate green).
phm.janet held the PersistentHashMap, the PersistentHashSet, AND the LazySeq
primitives — a lazy sequence has nothing to do with hash maps; both were just
tagged tables, which is why they shared a file (jolt-bvek). An agent looking for
lazy-seq realization would never grep phm.janet.
Move the LazySeq section (lazy-seq?/make-lazy-seq/realize-ls/ls-first/ls-rest/
ls-rest-cached/ls-seq/ls-count/lazy-cons) to a new self-contained lazyseq.janet
(janet builtins only, no jolt deps). Importers that used the fns through
(use ./phm) add (use ./lazyseq); host_interop's one phm/lazy-seq? becomes
lazy-seq?. Behaviour unchanged (covered by test/unit/lazy-seq-test.janet + the
full gate). phs split is a follow-up.
Co-authored-by: Yogthos <yogthos@gmail.com>
Cheap-version consolidation of the host-interop sprawl (jolt-jx5l). The JVM
class/method shims were scattered across javatime.janet, evaluator.janet,
api.janet and core.janet; the recent hiccup/markdown/malli fixes landed ad hoc.
- Rename javatime.janet -> host_interop.janet. The name described ~20% of its
contents (java.io/util/net/sql/lang all lived there); it is now the one home
for host shims, and greppable.
- Move the four hardcoded static tables (Math/Thread/System/Long) out of the
evaluator and register them through the generic class-statics registry. The
special-case dispatch in resolve-sym is deleted — one mechanism, not two.
- Move the collection-interop wiring (set-coll-realizer!, set-coll-interop!,
malli's LazilyPersistentVector/PersistentArrayMap statics) from api.janet into
host_interop's install-collections!. api's 40 lines of wiring become a single
(import ./host_interop).
No behavior change. Full per-package src/jolt/interop/ split is the follow-up;
string/number/object method tables and core's File/JDBC ctors stay put for now
(they're coupled to the dot-dispatch / collection layer). Full gate green.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Protocol/interop fixes to run metosin/malli
Bringing up malli (schema validation) surfaced a batch of protocol and host-interop
gaps. m/validate now works across the schema vocabulary (predicates, :map incl.
nested/optional, :vector, :tuple, :enum, :maybe, :and, bounded int/string).
- extend-type and reify now accept MULTIPLE protocols in one form (each bare
symbol switches the current protocol). reify records every protocol it
implements, so instance?/satisfies? recognise all of them.
- Protocol method params support destructuring: reify/extend-type/deftype/
defrecord emit (fn ...) (which desugars patterns) instead of raw fn*.
- instance? of a PROTOCOL works like satisfies? for reify/record instances,
matching short names across qualified/bare protocol references.
- @x reads as the qualified clojure.core/deref, so it still derefs where a ns
excludes and rebinds deref (malli does). Updated reader-test + the reader
spec/grammar (S11, deref rule).
- Java collection interop on jolt collections: .nth/.count/.valAt/.get/.seq/
.containsKey route to the clojure.core equivalent (1-arg and 0-arg paths).
- java.util.HashMap capacity/load-factor constructors + .putAll.
- A class used as a value resolves to its instances' type, so Pattern -> the
regex type (malli keys class-schemas by it).
- Shims for malli's load path: LazilyPersistentVector/createOwning and
PersistentArrayMap/createWithCheck statics.
m/explain not yet working (jolt-fjb1). Full gate green.
* satisfies? recognizes reify, consistent with instance?
A reify's protocol methods are instance-local, so they aren't in the global type
registry that type-satisfies? consults — satisfies? returned false for a reify
even when it implemented the protocol. Check the protocols the reify records on
itself (the same :jolt/protocols list instance? uses), matching short names like
instance? does. Covers single- and multi-protocol reify.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add architecture refactor plan
Synthesizes a six-part architectural review into phased, gate-validated cleanup
work. Targets LLM-maintainability: one home per feature, no god-files, explicit
checked contracts, no copy-paste dispatch. No code changes yet — the plan only.
* Refactor phase 0: dead code + isolated bugs
Pure cleanup ahead of the structural phases (docs/architecture-refactor-plan.md).
No behavior change except the two bug fixes, which are covered by a regression row.
Dead code (all verified zero-reference or overridden):
- core-resolve / core-satisfies? / core-type->str seed stubs + bindings —
resolve and satisfies? are interned by install-stateful-fns! (the seed copies
were shadowed); type->str was an inert SCI stub with no callers.
- find defined twice in 20-coll.clj; the dead copy returned a plain vector
(wrong — the live def at :787 returns a real map-entry) with a comment that
contradicted it.
- mark-hint (passes.clj), phs-to-struct (phm), shape-vals / ns-imports-fn
(types) — unreferenced.
- redundant local pad2 in javatime (module-level one already in scope).
Bugs:
- File.toURL stored :url but every :jolt/url method reads :spec, so a URL from
(.toURL file) returned nil from all its methods. Now stores :spec (+ spec row).
- pl-rest had a no-op (if (plist? r) r r); collapsed to r.
- :map-shapes? was missing from the deps-image cache key — two runs differing
only in map-shapes could reuse each other's image.
Also dropped read-quote's unused pos param. Full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Bringing up yogthos/markdown-clj surfaced a batch of Clojure-conformance gaps:
- clojure.java.io/writer returned nil for a Writer/StringWriter (it only
handled paths); now passes a Writer or file handle through, like reader does.
- StringWriter had no :close field, so with-open errored closing it.
- java.io.Reader had no .readLine method (only the :read-line-fn used by
line-seq); markdown's main loop calls .readLine directly.
- Writer.write(int) wrote the int's digits instead of the char for that code.
StringBuilder.append(int) keeps Java semantics (the digits) — the two differ,
so the char-code path is local to the writer, not shared render-piece.
- drop-while over a string errored in array/slice; it now char-seqs the string
like take-while/remove already do.
- re-seq returned an empty seq instead of nil on no match, so
(if-let [m (re-seq ...)] ...) always took the truthy branch — an infinite loop
in markdown's thaw-string.
- The #() reader didn't scan % inside map {} or set #{} literals, so
#(identity {:text %}) compiled as a 0-arg fn.
re-seq-nil and the #() map/set scan are general bugs, not markdown-specific.
Co-authored-by: Yogthos <yogthos@gmail.com>
