main.janet held ~45 lines of env-knob policy (open-mode / direct-link / optimize
/ shapes / whole-program gates) that couldn't be unit-tested without the CLI, and
two disk-image caches (api/init-cached, main/deps-image) each hand-built a
POSITIONAL "%q|%q|..." key that silently misaligned if a ctx-shaping knob was
added in only one place.
config.janet now owns:
ctx-shaping-env-vars the canonical list of env vars that shape the built ctx
ctx-cache-key a labeled key (name=value) over a prefix + every shaping
var, so adding a knob updates BOTH cache keys at once and
can't positionally alias two different builds
resolve-run-mode [open-mode? main-entry?] -> the ctx env knob map
main shrinks to: compute open-mode?/main-entry? from argv, call resolve-run-mode,
install the knobs. Both deps-image-path (main) and image-cache-path (api) build
their keys via ctx-cache-key. New test/unit/config-test.janet locks in the
run-mode cases and asserts every ctx-shaping env var participates in the key.
Scope: this is 4a + the 4b cache-key footgun fix. The optional 4b cleanup
(folding the load/save image dance + aot marshal helpers into one ctx_image
module) is left for a follow-up — it's lower value and higher blast radius.
No behavior change (cache keys now key on a superset of env vars, so at worst a
one-time cold rebuild). Gate green: conformance 355x3, clojure-test-suite 4718
pass (>= 4695 baseline), config-test, full jpm test exit 0.
types.janet held five concerns under one generic name. Split into sibling
modules along its existing section boundaries:
types_symbols characters + symbol helpers
types_var Var
types_ns Namespace
types_ctx Context (+ inst/uuid values)
types_protocols protocol/type registry + shape-records
types.janet is now a pure aggregator: it loads the clusters in dependency order
and re-exports their defs (import :prefix "" :export true), so every consumer
keeps its single (use ./types) unchanged.
Order-preserving (statically verified: zero backward references, zero
cross-cluster private helpers — the cleanest of the seed splits). No behavior
change.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
evaluator.janet was a 2597-line file with a 680-line eval-list. Split into
cluster modules behind a re-export aggregator (same pattern as core):
eval_base forward vars, syntax-quote, ns-loading, registries, jolt-invoke
eval_resolve symbol/var resolution, params, destructuring, class lookup
eval_runtime protocols, multimethods, deftype/reify, install-stateful-fns!
eval_special the special forms (eval-list dispatch)
evaluator.janet stays the module every consumer imports: it loads the clusters
in dependency order and re-exports their defs (import :prefix "" :export true),
so the five (use ./evaluator) consumers are unchanged. It still owns the
eval-form entry that ties resolution + special forms + map/coll evaluation.
In eval_special, the giant eval-list match is exploded: each multi-line arm is
now a named (defn eval-<form> [ctx bindings form] ...) — eval-def, eval-fn*,
eval-let*, eval-loop*, eval-try, eval-set!, eval-dot, etc. — and eval-list is a
thin dispatch table over them. "where is try handled" is now `grep eval-try`.
Order-preserving (statically verified: no symbol used before its cluster loads;
zero backward refs). 27 helpers shared across clusters are now public so `use`
shares them. The two near-duplicate .method dot blocks are NOT merged here — that
is a behavior-sensitive dedup tracked separately (jolt-eos3); this PR is pure
moves + the mechanical eval-list explosion, no behavior change.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
core.janet was a 3013-line grab-bag. Split into six order-preserving cluster
modules:
core_types vector helpers, predicates, math, comparison, equality
core_coll collections, transducers, seqs, HOFs, constructors
core_print string + pr-str/str rendering
core_io I/O, files, JDBC, compare, type
core_refs arrays, bit ops, coercions, hash, atoms/refs
core_extra additional clojure.core fns, transients, hashing
core.janet stays the module everyone imports: it loads the clusters in
dependency order and re-exports each one's defs (import :prefix "" :export true),
so every consumer keeps its single (use ./core) with no change. The aggregator
also owns core-bindings and init-core!, which reference fns from every cluster.
The split preserves definition order exactly (verified: no symbol is used in a
cluster that loads before its definition), so seed load-order semantics are
unchanged. Three private helpers used across clusters (map-entries-of,
map-assoc1, str-render-one) are now public so `use` shares them; the five
forward vars (canon-key/jolt-equal?/pr-render/core-compare/print-method-cb) each
stay within their owning cluster.
No behavior change. Gate green: conformance 355x3, clojure-test-suite 4718 pass
(>= 4695 baseline), full jpm test exit 0.
passes.clj was a 1486-line grab-bag mixing three weakly-coupled concerns. Split
along the clusters the review mapped (only run-passes + the dirty flag were
shared):
jolt.passes.fold const-fold + the shared scalar-const? predicate (base)
jolt.passes.inline inline + flatten-lets + scalar-replace
jolt.passes.types collection-type inference + success checker + driver API
jolt.passes façade: run-passes + :refer re-exports of the driver fns
the back end looks up by name
scalar-const? was used by both the inline pass and the inference walk, so it
moves to fold (the base layer) and both refer it. The check-mode state stays
private to jolt.passes.types behind a new run-inference fn; run-passes calls it.
build-compiler! loads the three in dependency order before the façade, mirroring
the existing jolt.ir -> jolt.analyzer bootstrap. No behavior change. Also fixed
the stale ns docstring that listed four passes and omitted the type system.
Gate green: conformance 355x3, clojure-test-suite 4718 pass (>= 4695 baseline),
full jpm test exit 0.
Completes the phm.janet decomposition (jolt-bvek): after lazyseq left, the set
follows. phm.janet is now purely the PersistentHashMap; phs.janet is the thin
set layer over it (members are keys -> true), and (use ./phm) for the builders.
Importers using set?/phs-* via (use ./phm) add (use ./phs); backend's emitted
set literal head changes phm/make-phs -> phs/make-phs. Behaviour unchanged
(sets verified interpreted + compiled; full gate green).
phm.janet held the PersistentHashMap, the PersistentHashSet, AND the LazySeq
primitives — a lazy sequence has nothing to do with hash maps; both were just
tagged tables, which is why they shared a file (jolt-bvek). An agent looking for
lazy-seq realization would never grep phm.janet.
Move the LazySeq section (lazy-seq?/make-lazy-seq/realize-ls/ls-first/ls-rest/
ls-rest-cached/ls-seq/ls-count/lazy-cons) to a new self-contained lazyseq.janet
(janet builtins only, no jolt deps). Importers that used the fns through
(use ./phm) add (use ./lazyseq); host_interop's one phm/lazy-seq? becomes
lazy-seq?. Behaviour unchanged (covered by test/unit/lazy-seq-test.janet + the
full gate). phs split is a follow-up.
Co-authored-by: Yogthos <yogthos@gmail.com>
Cheap-version consolidation of the host-interop sprawl (jolt-jx5l). The JVM
class/method shims were scattered across javatime.janet, evaluator.janet,
api.janet and core.janet; the recent hiccup/markdown/malli fixes landed ad hoc.
- Rename javatime.janet -> host_interop.janet. The name described ~20% of its
contents (java.io/util/net/sql/lang all lived there); it is now the one home
for host shims, and greppable.
- Move the four hardcoded static tables (Math/Thread/System/Long) out of the
evaluator and register them through the generic class-statics registry. The
special-case dispatch in resolve-sym is deleted — one mechanism, not two.
- Move the collection-interop wiring (set-coll-realizer!, set-coll-interop!,
malli's LazilyPersistentVector/PersistentArrayMap statics) from api.janet into
host_interop's install-collections!. api's 40 lines of wiring become a single
(import ./host_interop).
No behavior change. Full per-package src/jolt/interop/ split is the follow-up;
string/number/object method tables and core's File/JDBC ctors stay put for now
(they're coupled to the dot-dispatch / collection layer). Full gate green.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Protocol/interop fixes to run metosin/malli
Bringing up malli (schema validation) surfaced a batch of protocol and host-interop
gaps. m/validate now works across the schema vocabulary (predicates, :map incl.
nested/optional, :vector, :tuple, :enum, :maybe, :and, bounded int/string).
- extend-type and reify now accept MULTIPLE protocols in one form (each bare
symbol switches the current protocol). reify records every protocol it
implements, so instance?/satisfies? recognise all of them.
- Protocol method params support destructuring: reify/extend-type/deftype/
defrecord emit (fn ...) (which desugars patterns) instead of raw fn*.
- instance? of a PROTOCOL works like satisfies? for reify/record instances,
matching short names across qualified/bare protocol references.
- @x reads as the qualified clojure.core/deref, so it still derefs where a ns
excludes and rebinds deref (malli does). Updated reader-test + the reader
spec/grammar (S11, deref rule).
- Java collection interop on jolt collections: .nth/.count/.valAt/.get/.seq/
.containsKey route to the clojure.core equivalent (1-arg and 0-arg paths).
- java.util.HashMap capacity/load-factor constructors + .putAll.
- A class used as a value resolves to its instances' type, so Pattern -> the
regex type (malli keys class-schemas by it).
- Shims for malli's load path: LazilyPersistentVector/createOwning and
PersistentArrayMap/createWithCheck statics.
m/explain not yet working (jolt-fjb1). Full gate green.
* satisfies? recognizes reify, consistent with instance?
A reify's protocol methods are instance-local, so they aren't in the global type
registry that type-satisfies? consults — satisfies? returned false for a reify
even when it implemented the protocol. Check the protocols the reify records on
itself (the same :jolt/protocols list instance? uses), matching short names like
instance? does. Covers single- and multi-protocol reify.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* Add architecture refactor plan
Synthesizes a six-part architectural review into phased, gate-validated cleanup
work. Targets LLM-maintainability: one home per feature, no god-files, explicit
checked contracts, no copy-paste dispatch. No code changes yet — the plan only.
* Refactor phase 0: dead code + isolated bugs
Pure cleanup ahead of the structural phases (docs/architecture-refactor-plan.md).
No behavior change except the two bug fixes, which are covered by a regression row.
Dead code (all verified zero-reference or overridden):
- core-resolve / core-satisfies? / core-type->str seed stubs + bindings —
resolve and satisfies? are interned by install-stateful-fns! (the seed copies
were shadowed); type->str was an inert SCI stub with no callers.
- find defined twice in 20-coll.clj; the dead copy returned a plain vector
(wrong — the live def at :787 returns a real map-entry) with a comment that
contradicted it.
- mark-hint (passes.clj), phs-to-struct (phm), shape-vals / ns-imports-fn
(types) — unreferenced.
- redundant local pad2 in javatime (module-level one already in scope).
Bugs:
- File.toURL stored :url but every :jolt/url method reads :spec, so a URL from
(.toURL file) returned nil from all its methods. Now stores :spec (+ spec row).
- pl-rest had a no-op (if (plist? r) r r); collapsed to r.
- :map-shapes? was missing from the deps-image cache key — two runs differing
only in map-shapes could reuse each other's image.
Also dropped read-quote's unused pos param. Full gate green.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Bringing up yogthos/markdown-clj surfaced a batch of Clojure-conformance gaps:
- clojure.java.io/writer returned nil for a Writer/StringWriter (it only
handled paths); now passes a Writer or file handle through, like reader does.
- StringWriter had no :close field, so with-open errored closing it.
- java.io.Reader had no .readLine method (only the :read-line-fn used by
line-seq); markdown's main loop calls .readLine directly.
- Writer.write(int) wrote the int's digits instead of the char for that code.
StringBuilder.append(int) keeps Java semantics (the digits) — the two differ,
so the char-code path is local to the writer, not shared render-piece.
- drop-while over a string errored in array/slice; it now char-seqs the string
like take-while/remove already do.
- re-seq returned an empty seq instead of nil on no match, so
(if-let [m (re-seq ...)] ...) always took the truthy branch — an infinite loop
in markdown's thaw-string.
- The #() reader didn't scan % inside map {} or set #{} literals, so
#(identity {:text %}) compiled as a 0-arg fn.
re-seq-nil and the #() map/set scan are general bugs, not markdown-specific.
Co-authored-by: Yogthos <yogthos@gmail.com>
Some Clojure libraries loop with the Java Iterator protocol — e.g. hiccup's
iterate! does (let [it (.iterator coll)] (while (.hasNext it) (f (.next it)))).
jolt had no Iterator, so (.iterator coll) returned nil and the loop did nothing
(silently dropping content). Add an (.iterator coll) object-method that returns a
:jolt/iterator over any seqable, with hasNext/next; the collection is materialized
via core's realize-for-iteration (late-bound through the evaluator since core
loads after it, wired in api). Found while bringing up weavejester/hiccup.
Co-authored-by: Yogthos <yogthos@gmail.com>
Two things made a program run catastrophically slow to start (ring-app: 111s),
which is backwards — direct-linking should make startup FASTER (Clojure: smaller
classes, faster startup; Stalin: whole-program analysis is strictly ahead-of-
time, zero startup cost). We were doing the opposite: paying compile-time work at
every startup.
1. Decouple inference from direct-linking. Direct-linking (cheap compile-time
call resolution) was bundled with the whole inference/specialization pipeline
(inlining + scalar replacement + structural types + the per-ns re-emit
fixpoint) — the expensive part. Now a program run direct-links + uses records
by default (fast, faster calls) but the inference is opt-in via JOLT_OPTIMIZE
(or an explicit JOLT_DIRECT_LINK / the native build). ring-app: 111s -> 6s.
2. Compile dependencies once. The baked core loads in ~10ms, but a program still
re-compiled every dependency namespace (reitit, ring, selmer, honeysql, …)
from source on every run — the whole 6s. Snapshot the ctx after the require
chain to a per-project image (the same marshal/fork the core image uses),
keyed on version + entry + source roots + flags, with a per-file mtime
manifest so any edit invalidates it. First run compiles + caches; later runs
fork the image (~10ms) and skip compilation. ring-app: 6s -> ~1s warm,
competitive with the JVM. JOLT_NO_DEPS_CACHE disables.
Full gate green (all -m integration tests now exercise the cache); deps-image
invalidation verified (touching a source recompiles).
Under direct-linking a record is a Janet tuple (its shape-rec), and core-vector?
just delegated to jvec? which is true for any tuple — so (vector? a-record) and
(sequential? a-record) returned true. That broke map-destructuring of a record:
the destructure coerce treats a sequential source as & {:keys} kwargs and does
(apply hash-map x), so destructuring a record fed its entries to make-phm as a
flat kv-list and corrupted. Surfaced as reitit's router crashing on a wildcard
route ('expected integer key for tuple in range [0,5), got 5') whenever the new
direct-link default was on; minimal repro is (let [{:keys [a]} (->R ...)] ...).
Fix: core-vector? excludes shape-recs, matching Clojure (a record is not a
vector or sequential). jvec? is unchanged for internal representation dispatch.
Regression cases added to record-declared-shape-test.
Running a program is a closed world — every namespace is required, then it runs
to completion — so make it direct-link by default (inlining, record shapes, the
inference's specialization), and for a -m/-M entry auto-enable the whole-program
cross-namespace inference pass. A decomposed multi-namespace program was ~3.7x
slower than the same code in one namespace purely because per-namespace
inference can't see a caller in a not-yet-loaded namespace; this closes that for
the common case with no flags and no hints.
Interactive modes (repl, -e, nrepl-server) stay indirect/open — they have to let
you redefine vars, which direct-linking seals against. Opt-outs:
JOLT_NO_DIRECT_LINK forces the open path even for a program run (hot-reload,
runtime redefinition); JOLT_NO_WHOLE_PROGRAM keeps direct-linking but per-ns;
JOLT_DIRECT_LINK / JOLT_WHOLE_PROGRAM still force-on. Namespaces required inside
-main (after the batch pass) fall back to per-ns inference.
The success checker (RFC 0006) rides on the inference for free, but a casual
program run shouldn't spam type warnings just because it now direct-links, so its
default-on is suppressed when direct-linking was auto-enabled (:direct-link-auto?);
an explicit JOLT_DIRECT_LINK or JOLT_TYPE_CHECK still turns it on. whole-program-
test and devirt-test opt their per-ns baseline out of the new auto-default.
Docs: RFC 0005 gains 'Compilation modes and defaults' + 'Cross-namespace
inference'; RFC 0004 documents cross-ns/param hints; self-hosting-compiler and
--help updated. Full gate green.
A ^RecordType hint only resolved against the current namespace's ctor key, so a
hint naming a record defined in another namespace degraded to :any. That made a
decomposed multi-namespace program much slower than the monolith: per-namespace
inference can't see a record param's callers in other namespaces, and the
declared hint that could have typed it was dropped.
Resolution now works cross-namespace, for both record FIELD hints (defrecord)
and fn PARAM hints, in both spellings — ^Vec3 where the type is referred and
^v/Vec3 where the namespace is aliased:
- reader keeps a tag's namespace qualifier (^t/Ray -> "t/Ray", was "Ray").
- make-deftype-ctor-impl indexes each ctor closure by value; record-hint-ctor-key
resolves a hint name against the COMPILE ns (referred names live there; aliases
resolve through it) and maps the type var's root back to its home ctor key.
Using the ctor value, not the var's :ns, is what makes :refer work — :refer
re-interns a fresh var whose :ns is the referring ns.
- the analyzer captures record param hints as arity :phints [name ctor-key];
reinfer-def seeds those param types, so a record param is typed even with no
inferred caller — the open-world / cross-ns case.
Effect on the multi-namespace ray tracer: per-ns compile 30.4s -> 7.9s with
param hints, matching whole-program (8.1s) and the single-ns monolith (8.3s).
cross-ns-hints-test covers field + param hints, refer + as, and the reader tag.
direct-var? now treats a cfunction root the same as a function root, so a
call/ref to a native fn (clojure.math/sqrt et al.) embeds the value instead of
a per-call cell deref. This was the hot indirection in the ray tracer — sqrt
runs every bounce — and it applies in every direct-link build, not just
whole-program.
const-link? is new and whole-program-only: in a closed world every non-dynamic
var has a stable root, so embed it as a constant (quoted unless it's already
callable) rather than reading the cell each reference. Covers what direct-var?
can't — ^:redef vars (reloading is off under the flag), data defs, and record
type/ctor roots. Dynamic vars stay indirect; a nil (not-yet-defined) root stays
indirect and the whole-program re-emit picks it up once the root is in place.
Measured on the records ray tracer: hot-path indirect refs (sqrt + data vars)
gone; the only indirect refs left are cold defrecord self-references. whole-
program-test now also checks a ^:redef fn and a data def so the per-ns vs
whole-program comparison guards const-link soundness.
A record field can carry a type hint — ^Vec3 (a defined record type) or ^:num —
and the inference now resolves it so reading the field back yields that exact type
instead of :any. A Vec3 stored in a Ray field reads out as Vec3, so the vec ops on
field-read values prove their reads (bare-index). This is Stalin's per-slot type
sets, but DECLARED rather than inferred: the exact shape is known up front.
- deftype captures each field's :tag / :num metadata (was stripped) and passes it
to make-deftype-ctor; the ctor registers per-field tags, resolving a record-type
hint to its ctor-key (same-ns) so the inference can look it up directly.
- call-ret-type builds a record's struct type with field types resolved from the
hints, recursing into nested record types (depth-bounded for self/cyclic types).
Measured: a nested-record read loop (:r (:origin ray)) runs 1.3s with ^Vec3 hints
vs 7.1s without — 5.5x. This is the lever the ray tracer needed (vecs flow through
container fields); records without it read back as :any and stay unproven.
A protocol method call compiles to (protocol-dispatch proto method this rest) — a
runtime registry walk (type-tag -> proto -> method) on every call, ~19x a direct
call. When the inference proves the receiver (arg 0) is a known record type, the
call now resolves to a DIRECT method call at compile time, skipping the registry.
- defprotocol registers each method's var-key 'ns/method' -> [proto method] (a
ctx-capturing register-protocol-methods! emitted into the do-block); infer-unit!
feeds it to the inference via a box (like record-shapes).
- the record-ctor return type carries :type (the record tag) so the inference
knows the receiver type; the :else invoke case annotates a protocol call whose
arg0 has a known :type with :devirt-{type,proto,method}.
- emit-invoke resolves the impl via find-protocol-method at emit time and emits a
direct call to the embedded impl fn value. Unknown/polymorphic receivers (no
proven :type) fall back to the dispatch path unchanged.
Measured: removes the dispatch overhead (14.7s -> 9.3s on a 10M-call loop); the
remaining cost is the method body itself (non-inlined, unproven reads) — inlining
the resolved method is the follow-up (jolt-t6r) toward direct-call speed.
Sound under the closed-world assumption direct-linking already makes (the impl is
resolved + embedded at compile time). Adds devirt-test (subprocess: dispatched ==
devirtualized across polymorphic dispatch, unknown-receiver fallback, and
heterogeneous collections). Stalin's compile-call/callee-environment is the model.
JOLT_WHOLE_PROGRAM (requires direct-linking) defers the per-namespace inference
and runs ONE fixpoint over every user unit at once, so param types propagate
across namespace boundaries — a non-inlined fn's record params get proven from
its callers in another unit, which the per-ns pass can't see. Sound only under
the closed-world assumption (no later eval/redefinition) the flag asserts; slow,
memory-heavy builds are the documented trade-off (the reason it's opt-in).
infer-unit! now takes one ns-name OR a list; infer-program! gathers all recorded
user namespaces and runs the existing fixpoint over the union (re-emit was already
ns-agnostic — keyed by var-key, callee-first). The evaluator defers + records each
unit under the flag; run-main triggers infer-program! after all requires, before
-main. Off by default — per-ns behaviour unchanged.
Measured: a recursive (non-inlined) cross-ns record reader runs 1.66x faster
(8.9s -> 5.3s) — params proven -> bare-index reads. NOTE: small accessor fns are
INLINED cross-ns and records carry GLOBAL declared shapes, so most record reads
are already proven without this pass; the win is for non-inlined hot fns, and it's
the foundation for future whole-program work (devirtualization, unboxing).
Adds whole-program-test (subprocess soundness: per-ns and whole-program produce
identical results on a cross-ns record program).
Records (defrecord/deftype) are now shape-recs in a direct-linking unit by
default — no JOLT_SHAPE flag. A record's shape is DECLARED, so the inference
proves field reads by a lookup, not fragile shape inference, and they bare-index.
Result: ~1.4x faster than the :jolt/deftype table form on a record-heavy loop
(3.9s vs 5.5s), driven by cheaper construction + proven bare-index reads.
Two gates now:
- :shapes? — shape-recs active; records use declared-shape layout + bare
index reads. On with direct-linking (where the inference runs).
- :map-shapes? — also shape generic const-key maps. Opt-in (JOLT_SHAPE), because
shaping maps net-loses on unproven reads (measured). Records win.
- call-ret-type types a record ctor (->Name) as a struct of its declared shape,
fed from a ctx-env registry populated at deftype; field reads on the result
bare-index. (set-record-shapes!/set-map-shapes! wired through infer-unit!.)
- sidx reads the field's position from the :shape vector AS-IS (declared order
for records, str-sorted for map literals) — no re-sort — so any field order
bare-indexes correctly. The map :map case only sets :shape under :map-shapes?.
- record-shape-for interns the descriptor per (type, fields), not per type: a
record redefined with different fields now gets a fresh descriptor instead of a
stale one (fixes redef descriptor staleness; old instances stay valid).
Adds record-declared-shape-test (declared-order reads, incl. non-alphabetical
fields, through fn boundaries + protocol method bodies). Known pre-existing edge
case filed as jolt-wf4 (direct (:f (->R …)) read returns nil after a record is
redefined with different fields; let-bound read works; repros without shapes).
Measurement (vec-heavy benchmark + read-mechanism micro-benchmarks) showed that
shape-rec'ing generic const-key maps is a net loss in a bytecode VM: an unproven
field read can't beat a native Janet struct-get (one opcode), and an inline cache
doesn't transfer to a VM jolt doesn't control. The win is real only for PROVEN
reads (bare-index beats struct) — i.e. construction-heavy or inference-friendly
code, not general map-through-fn code.
So shapes are opt-in again (JOLT_SHAPE), not defaulted on in direct-link. Kept the
get-or-shape fix: the non-proven shape read now indexes inline off the descriptor
instead of calling core-get (which was ~2.5x slower per read). :shapes? is the
single opt-in gate; image cache keys on JOLT_SHAPE + JOLT_NO_SHAPE.
Next: records with declared shapes get fast field access by default (the general
case), which is where the proven-read win actually lands.
Record shape-recs now compare type-aware like the table form: eq-map-pairs
returns nil for a record so equality falls to deep=, which is type-aware via the
per-type interned descriptor. A record equals only a same-type record, never a
plain map (cross-type and record-vs-map were wrongly equal under JOLT_SHAPE).
assoc of a new (undeclared) key keeps the record type and grows a slot, matching
Clojure's record-extension semantics.
Adds test/integration/record-shape-test.janet locking in the runtime record
mechanism (tag, field access, virtual :jolt/deftype, type-preserving assoc,
dissoc demotion, type-aware equality, #ns.Type{...} printing).
Records (defrecord/deftype) become shape-recs whose descriptor also carries
:type, under JOLT_SHAPE (flag off keeps the :jolt/deftype table form). A record
is a Janet tuple [descriptor field0 ...] in declared field order, so the
positional ->Name ctor maps args straight to slots.
- record-tag unifies the type accessor over both reps; instance?/satisfies?/
protocol dispatch and the . interop path go through it.
- virtual :jolt/deftype key on record shape-recs (via shape-get) keeps every
existing (get obj :jolt/deftype) dispatch site working.
- emit-kw-lookup's non-proven fallback routes any tuple to core-get (shape
aware), not gated on compile-time JOLT_SHAPE — core is baked without the flag
but still receives user shape-recs.
- assoc keeps the type (same-shape in place, new key grows a slot Clojure-style);
dissoc of a declared field demotes to a plain map; pr prints #ns.Type{...}.
- JOLT_SHAPE added to the image-cache key (it shapes core's own compiled paths).
The inference dropped the complete :shape whenever it rebuilt a struct type
(cap) or joined two (join-t/merge-fields), so a vec3 retrieved from a container
or a fn param typed across call sites lost its layout and every field read fell
to the slow descriptor path. Two fixes:
- cap preserves :shape: capping truncates field VALUES below the depth limit but
never the key SET, so the layout is still complete. It also recurses into
fields, so a shaped value nested in a container (a vec3 inside a hit-info)
keeps its own :shape — which is what lets (:r (:normal hit-info)) bare-index.
- join-t preserves :shape when both sides are the SAME complete shape (the
merged struct has the same keys); different shapes still drop it. This carries
the shape through if-joins and the inter-procedural fixpoint's call-site joins.
Result: the ray tracer goes from 22s (R1, correct-but-descriptor-path) to 4.36s
— 2.7x FASTER than the 11.7s no-shape baseline, and ~3x the JVM (was 8.5x), with
byte-identical output. The compounding of cheaper tuple construction plus
bare-index reads across the whole render far exceeds the per-op estimate.
Gate green flag-off, suite 4718, default-path bench even, transparency intact.
Removes ALL hardcoding. Every constant-key map literal in user code becomes a
shape-rec (a Janet tuple [descriptor v0 v1 ...]); the descriptor is interned
per key SET with a single canonical key order owned by the runtime
(types/shape-sort), so every site that builds or reads a shape agrees.
Consistency is the foundation: shape-recs are made UNCONDITIONALLY for
const-key user maps (emit-map), not gated on the inference — so a value's
representation always matches what the type system claims, which was the bug
that broke the earlier generalization (a ray built as a struct but bare-indexed
as a shape). Gated on :inline? so it applies to user data only, never to core
or the compiler's own IR-node maps.
Transparency layer (the shape-rec block now lives in types.janet, reachable by
core + evaluator + backend): get, assoc, dissoc, count, contains?, map?, first,
seq, the central realize-for-iteration normalizer (keys/vals/reduce-kv), eq-map-
pairs + eq-seqable (equality), pr-render (print), jolt-call (compiled IFn), and
the interpreter's coll-lookup all handle shape-recs. A new spec
(shape-transparency-test) asserts each op matches the equivalent struct map,
including nil/false values (which shape-recs store positionally — unlike
structs).
The ray tracer renders byte-identically (mean 122.04). Gate green with the flag
off, suite 4718. NOT yet faster — every field read currently takes the
descriptor path because the inference drops the complete :shape through joins
and containers; that's Round 2 (completeness preservation). All behind
JOLT_SHAPE (off by default).
Removes the {:r :g :b} hardcoding. ANY constant key set is now a shape:
- inference: a struct type from a map LITERAL carries :shape (its canonical
str-sorted key vector — completeness); joins/access-inferred structs lack
it, so they never get a bare index. The literal node and lookup subjects
carry the shape; the back end derives the index from it.
- backend: emit-map turns any shape-tagged const-key map into a shape tuple;
emit-kw-lookup reads the field by bare index when the complete shape is
proven, else by the value's own descriptor (so a shape-rec whose :shape was
dropped by a join still reads correctly).
- runtime: core-get and core-assoc handle shape-recs.
Status: CORRECT for direct field access, container round-trips, and assoc
(minimal repros pass). NOT yet complete — the full ray tracer still hits an
uncovered path (a shape-rec reaching a map op without coverage: keys/vals/
count/seq/equality/print/jolt-call/dissoc/contains?/the interpreter's
coll-lookup all still need shape-rec branches). And the perf win needs
COMPLETENESS PRESERVATION through joins/containers (merge-fields/cap drop
:shape today, so nested vec3 access falls to the descriptor path, slower than
a struct get) — without it the general version is slower than the vec3
prototype.
All behind JOLT_SHAPE (off by default). Gate green with the flag off, suite
4718. This preserves the general design; the transparency layer + completeness
preservation are the remaining multi-session work.
Validated prototype of the hidden-class object-model change. A vec3-shaped
{:r :g :b} map literal is represented as a cheap Janet tuple [shape vb vg vr]
instead of a struct (~2x cheaper to construct); a lookup on a value the
inference PROVES is the shape reads by bare index with no runtime check.
Result on the ray tracer (direct-link): 12.3s -> 10.7s (~13% faster), with
byte-identical pixel output. The shape value flows transparently through
hit-info/ray/material containers and the colors vector; core-get handles it
(inline check, no fn call) so an unspecialized access is still correct.
Key lessons baked in: the lookup MUST compile to a bare index (a runtime
shape check, even inlined, taxed every field read and made it 2.5-3.4x
SLOWER) — so the inference gained a :shape hint (struct type with keys
exactly {:r :g :b}) that the back end turns into (in m idx). The descriptor
is quoted when embedded (its keys are a parens tuple Janet would otherwise
try to CALL).
All behind JOLT_SHAPE (off by default). Gate green, suite 4718, default-path
bench even. Scoped to the one shape; NOT yet sound in general (assumes every
vec3-shaped value is a shape-rec, true under the flag for the ray tracer) nor
fully transparent (only core-get + the inlined lookup; jolt-call/equality/
print/keys not yet covered). Those are the next steps toward a real feature.
Pivot from a jolt reimplementation to running the upstream library verbatim.
Vendors the real clojure/tools/logging.clj; jolt provides the backend and the
host primitives it needs. Language features (broadly useful for real Clojure
libs), all covered in 3-mode conformance + spec suites:
- defmacro: multi-arity dispatch (jolt-q8l) and a docstring + attr-map + params
head (jolt-qnr) — the 4-arity log macro and every level macro need these.
- syntax-quote resolves an alias-qualified symbol to its target ns (jolt-9av),
so a macro template (impl/get-logger) resolves at the use site.
- the ns macro unwraps ^{:map} metadata on the ns name (jolt-8w2 workaround,
matching def/defn/defmacro).
- a namespace object self-evaluates, so ~*ns* can be spliced into a template.
Host shims (ported from / modeled on clojure where applicable):
- clojure.string/trim-newline (ported, CharSequence interop -> count/subs)
- agent/send-off/send (minimal synchronous stubs; jolt has no thread pool/STM)
- clojure.lang.LockingTransaction/isRunning -> false
- a minimal clojure.pprint (pprint/with-pprint-dispatch/code-dispatch, for spy)
- clojure.tools.logging.impl: a jolt stderr LoggerFactory backend (the library's
designed pluggable extension point)
docs/libraries.md lists tools.logging; grammar.ebnf metadata note clarified.
Conformance 355/355 x3 modes; full jpm test gate green.
Two try bugs that blocked migratus's migrate (run/table-exists? are multi-body
try/catch/finally with janet-interop jdbc calls, which punt to the interpreter):
- The interpreter's try took only form 1 as the body, dropping later body forms
before the clauses, and did not run finally on the success path when a catch
was present. Rewritten to collect every body form before the first
catch/finally and to always run finally (via defer, so it fires even if a
catch body throws).
- current-ns leaked after a caught throw from an INTERPRETED fn (it sets ns to
its defining ns and can't restore on unwind). The interpreter's try now
restores; the compiled try (emit-try) snapshots the ns at entry and resets it
in the catch via new __current-ns/__set-current-ns! core fns. Statement
values also get a :close key so with-open can close them.
With these, migratus migrate/rollback round-trips on jolt (verified end to end
against sqlite). Conformance 335/335 x3, full gate green.
methods/get-method now take the multimethod VALUE (Clojure semantics), so the
arg must resolve to compile. The isolated analyzer never defines mf, so point
these two at print-method (a real defmulti) instead.
File API (jolt-hjw): io/file and (File. …) build a tagged :jolt/file value
(instance? File true) with a full method surface (isFile/isDirectory/exists/
getName/getPath/getAbsolutePath/listFiles/toPath/delete/createNewFile/…) backed
by os/ and file/. file-seq is File-aware (leaves are File values). str/slurp/spit
coerce :jolt/file to its path. ClassLoader/getSystemClassLoader + a classloader
stub whose getResource returns nil degrade migratus's classpath lookup to the
filesystem. java.nio.file Path/FileSystem/PathMatcher are shimmed just enough for
script-excluded?'s glob (recursive * / ? matcher).
Three bugs found getting migratus's migration discovery to work:
- (assoc nil k v) returned a raw janet table, not a map, so assoc-in built tables
that count/seq rejected. Now returns an immutable map.
- methods/get-method resolved the multimethod symbol at runtime in the current
ns, so a bare multifn ref in its defining ns saw an empty table once defmethods
lived elsewhere. Now they take the multimethod VALUE and recover the var via a
registry (Clojure semantics).
- defmulti now drops a leading docstring/attr-map (migratus's multimethods carry
docstrings) instead of treating the docstring as the dispatch fn.
Conformance 335/335 x3, clojure-test-suite at baseline.
The analyzer always took (nth items 2) as the value, so (def x "doc" 42)
bound x to the docstring and dropped 42. Now it mirrors the interpreter:
when there are 4+ items and item 2 is a string, item 2 is the docstring
(attached as :doc meta) and item 3 is the value. Conformance 335/335 x3.
For the migratus next.jdbc shim (jolt-0z5):
- core.janet: __jdbc-wrap-conn / __jdbc-conn-raw / __jdbc-make-stmt builtins.
A connection is a tagged wrapper over a jdbc.core conn carrying a clj :exec
callback so the host Statement.executeBatch runs SQL without a janet->clj call.
- javatime.janet: tagged-methods for :jolt/jdbc-conn (setAutoCommit/isClosed/
close/getMetaData), :jolt/jdbc-meta (getDatabaseProductName), :jolt/jdbc-stmt
(addBatch/executeBatch/close); java.sql.Timestamp ctor -> millis.
- evaluator.janet: instance? case for Connection/java.sql.Connection so
migratus's do-commands runs SQL through its Connection branch.
Two defn/defmacro fixes found loading migratus.core (both rooted in the reader
representing ^{:map} name metadata as a with-meta form, jolt-8w2):
- defmacro special form: unwrap a with-meta name (mirrors def), and handle the
arity-clause form (defmacro name ([params] body...)) like fn/defn — a params
vector reads as a tuple, an arity clause as a list (array).
- defn overlay: pass the bare (unwrapped) name to fn while def keeps the meta.
Conformance 335/335 x3 modes.
