simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: general shape-record mechanism — consistent representation + transparency (jolt-t34 R1)

Browse source 
Removes ALL hardcoding. Every constant-key map literal in user code becomes a
shape-rec (a Janet tuple [descriptor v0 v1 ...]); the descriptor is interned
per key SET with a single canonical key order owned by the runtime
(types/shape-sort), so every site that builds or reads a shape agrees.

Consistency is the foundation: shape-recs are made UNCONDITIONALLY for
const-key user maps (emit-map), not gated on the inference — so a value's
representation always matches what the type system claims, which was the bug
that broke the earlier generalization (a ray built as a struct but bare-indexed
as a shape). Gated on :inline? so it applies to user data only, never to core
or the compiler's own IR-node maps.

Transparency layer (the shape-rec block now lives in types.janet, reachable by
core + evaluator + backend): get, assoc, dissoc, count, contains?, map?, first,
seq, the central realize-for-iteration normalizer (keys/vals/reduce-kv), eq-map-
pairs + eq-seqable (equality), pr-render (print), jolt-call (compiled IFn), and
the interpreter's coll-lookup all handle shape-recs. A new spec
(shape-transparency-test) asserts each op matches the equivalent struct map,
including nil/false values (which shape-recs store positionally — unlike
structs).

The ray tracer renders byte-identically (mean 122.04). Gate green with the flag
off, suite 4718. NOT yet faster — every field read currently takes the
descriptor path because the inference drops the complete :shape through joins
and containers; that's Round 2 (completeness preservation). All behind
JOLT_SHAPE (off by default).
This commit is contained in:
Yogthos 2026-06-13 17:49:45 -04:00
parent e377c223b6
commit c6b964c1b1
5 changed files with 153 additions and 49 deletions
Download patch file Download diff file Expand all files Collapse all files

29
src/jolt/backend.janet
View file

@ -370,11 +370,13 @@
"(a phm/sorted/transient/lazy-seq), not the plain "
"struct/record the ^:struct/^Record hint asserts")]))
# Subject carries a complete :shape (jolt-t34) => it is provably a shape-rec;
# the field reads by bare index. The shape is the inference's canonical key
# vector, so the index is the key's position in it, +1 for the descriptor.
# the field reads by bare index. The :shape is just the KEY SET; the layout
# comes from the runtime's canonical shape-sort (the same order shape-for and
# emit-map use), so construction and lookup always agree.
(def sidx
(when (and (os/getenv "JOLT_SHAPE") subj-node (subj-node :shape))
(def sk (let [s (subj-node :shape)] (if (pv/pvec? s) (pv/pv->array s) s)))
(def raw (let [s (subj-node :shape)] (if (pv/pvec? s) (pv/pv->array s) s)))
(def sk (shape-sort raw))
(var pos nil) (var i 0)
(each kk sk (when (= kk k) (set pos i)) (++ i))
(when pos (+ pos 1))))
@ -487,15 +489,20 @@
(unless (and (= :const (k :op))
(or (keyword? kv) (string? kv) (number? kv) (boolean? kv)))
(set fast false)))
# Shape-record fast path (jolt-t34, JOLT_SHAPE): a const-key map literal the
# inference tagged with a complete :shape becomes a shape tuple (≈2x cheaper
# than a struct). The shape is the inference's canonical (str-sorted) key
# vector — ANY constant key set, no hardcoding. Values are emitted in that
# order; element 0 is the compile-time shape descriptor.
# Shape-record representation (jolt-t34, JOLT_SHAPE): EVERY constant-key map
# literal becomes a shape tuple (≈2x cheaper than a struct), CONSISTENTLY —
# not gated on the inference, so a value's representation always matches what
# the type system claims about it. The layout is the runtime's canonical
# shape-sort of the keys (the single source of truth all sites share). Values
# are emitted in that order; element 0 is the compile-time shape descriptor.
# gated on :inline? — shapes apply to USER code only, never to core or the
# compiler's own IR-node map literals (which the back end reads with raw
# keyword access and would break if turned into tuples)
(def shape-keys
(when (and fast (os/getenv "JOLT_SHAPE") (node :shape))
(def sk (node :shape))
(if (pv/pvec? sk) (pv/pv->array sk) (if (or (tuple? sk) (array? sk)) sk nil))))
(when (and fast (os/getenv "JOLT_SHAPE") (get (ctx :env) :inline?))
(def ks @[])
(each pair pairs (array/push ks ((norm-node (in (vview pair) 0)) :val)))
(shape-sort ks)))
(if shape-keys
(do
(def desc (shape-for shape-keys))

54
src/jolt/core.janet
View file

@ -122,6 +122,7 @@
(cond
# nil is an empty seq in Clojure — iterating it yields nothing.
(nil? c) @[]
(shape-rec? c) (map (fn [k] (tuple k (shape-get c k nil))) (shape-keys c))
(pvec? c) (pv->array c)
(plist? c) (pl->array c)
(set? c) (phs-seq c)
@ -208,7 +209,8 @@
# Tagged structs (symbols/chars/uuids — anything with :jolt/type) are VALUES,
# not maps. (sorted-map? is defined later, so the table check is inlined.)
(defn core-map? [x]
(or (phm? x)
(or (shape-rec? x)
(phm? x)
(and (struct? x) (nil? (get x :jolt/type)))
(and (table? x)
(or (not (nil? (get x :jolt/deftype)))
@ -311,6 +313,8 @@
an array; otherwise nil. Lets = compare across tuple/array/lazy-seq."
[x]
(cond
# a shape-rec is a MAP, not a sequence, even though it is a tuple
(shape-rec? x) nil
(lazy-seq? x) (realize-for-iteration x)
(pvec? x) (pv->array x)
(plist? x) (pl->array x)
@ -322,6 +326,7 @@
"Return [k v] pairs for a map-like value (phm/sorted-map/struct/table), else nil."
[x]
(cond
(shape-rec? x) (map (fn [k] @[k (shape-get x k nil)]) (shape-keys x))
(phm? x) (phm-entries x)
# sorted-map equals any map with the same pairs (representation-agnostic, as
# in Clojure); sorted-set is handled by the set branch of jolt-equal?
@ -495,42 +500,6 @@
(error "Vector arg to map conj must be a pair")))
result)))))))))))))
# --- shape records (hidden classes, jolt-t34) -------------------------------
# A "shape record" is a cheap fixed-layout representation for a map literal
# whose keys are a known compile-time set (e.g. a vec3 {:r :g :b}). It is a
# Janet tuple [SHAPE v0 v1 ...] where SHAPE is an interned descriptor struct
# {:jolt/shape KEYS :idx {k->pos}}. Construction is a tuple (≈2x cheaper than a
# struct), const-keyword lookup compiles to an index, and general map ops below
# recognize it via shape-rec? and treat it as a map — so it is transparent
# wherever it flows. Created only by the backend when JOLT_SHAPE is on and the
# inference proves the shape; the runtime support here is always present so a
# shape value is handled correctly regardless.
(def- shape-cache @{}) # sorted-keys-tuple -> interned shape descriptor
(defn shape-for
"Interned shape descriptor for an ordered key vector (keys in layout order)."
[keyv]
(def kk (tuple ;keyv))
(or (get shape-cache kk)
(let [idx @{}]
(var i 0) (each k keyv (put idx k i) (++ i))
(def desc (struct :jolt/shape kk :idx (table/to-struct idx)))
(put shape-cache kk desc)
desc)))
(defn shape-rec? [x]
(and (tuple? x) (> (length x) 0)
(struct? (in x 0)) (not (nil? (in (in x 0) :jolt/shape)))))
(defn shape-keys [rec] ((in rec 0) :jolt/shape))
(defn shape-get [rec k default]
(def pos (get ((in rec 0) :idx) k))
(if (nil? pos) default (in rec (+ pos 1))))
(defn shape-assoc [rec k v]
# assoc on a known key keeps the layout; a new key falls back to a struct
(def desc (in rec 0))
(def pos (get (desc :idx) k))
(if (nil? pos)
(let [t @{}] (each kk (desc :jolt/shape) (put t kk (shape-get rec kk nil))) (put t k v) (table/to-struct t))
(let [a (array ;rec)] (put a (+ pos 1) v) (tuple ;a))))
(defn core-assoc [m & kvs]
(when (odd? (length kvs))
(error "assoc expects an even number of key/value arguments"))
@ -589,6 +558,8 @@
(defn core-dissoc [m & ks]
(cond
(nil? m) nil
# dissoc loses a key -> the shape changes; bridge through a struct (cold op)
(shape-rec? m) (core-dissoc (shape->struct m) ;ks)
(core-sorted-map? m) ((sorted-op m :dissoc) m ks)
(phm? m) (do (var result m) (each k ks (set result (phm-dissoc result k))) result)
# reject clearly non-map values (scalars, sequences, sets, symbol/char structs)
@ -633,6 +604,7 @@
(defn jolt-call [f & args]
(cond
(or (function? f) (cfunction? f)) (apply f args)
(shape-rec? f) (core-get f (get args 0) (get args 1))
(keyword? f) (core-get (get args 0) f (get args 1))
(and (struct? f) (= :symbol (f :jolt/type))) (core-get (get args 0) f (get args 1))
(core-sorted? f) ((sorted-op f :get) f (get args 0) (get args 1))
@ -673,6 +645,7 @@
# get-in now lives in the Clojure collection tier (core/20-coll.clj).
(defn core-contains? [coll key]
(if (shape-rec? coll) (shape-contains? coll key)
(if (core-sorted? coll) (if ((sorted-op coll :contains) coll key) true false)
(if (core-transient? coll)
(case (coll :kind)
@ -685,7 +658,7 @@
(if (table? coll) (not (nil? (coll key)))
(if (or (tuple? coll) (array? coll))
(and (number? key) (>= key 0) (< key (length coll)))
false)))))))))
false))))))))))
# Coerce a Clojure IFn value to a Janet-callable fn for higher-order fns
# (map/filter/sort-by/group-by/...). Janet functions pass through; a keyword or
@ -708,6 +681,7 @@
(defn core-count [coll]
(cond
(nil? coll) 0
(shape-rec? coll) (shape-count coll)
(core-transient? coll) (length (if (= :vector (coll :kind)) (coll :arr) (coll :tbl)))
(core-sorted? coll) ((sorted-op coll :count) coll)
(lazy-seq? coll) (ls-count coll)
@ -722,6 +696,7 @@
(defn core-first [coll]
(cond
(shape-rec? coll) (core-first (shape->struct coll))
(core-sorted? coll) ((sorted-op coll :first) coll)
(lazy-seq? coll) (ls-first coll)
(pvec? coll) (if (= 0 (pv-count coll)) nil (pv-nth coll 0))
@ -800,6 +775,7 @@
(defn core-seq [coll]
(cond
(shape-rec? coll) (tuple ;(map (fn [k] (tuple k (shape-get coll k nil))) (shape-keys coll)))
(core-sorted? coll) ((sorted-op coll :seq) coll)
(or (nil? coll) (and (or (tuple? coll) (array? coll)) (= 0 (length coll)))) nil
# Cell-based emptiness, NOT (nil? (ls-first)): a lazy-seq whose first element
@ -1655,6 +1631,8 @@
(buffer/push-string buf (ns-display-name v))
(buffer/push-string buf "]"))
(and (table? v) (= :jolt/var (get v :jolt/type))) (buffer/push-string buf (var-display v))
(shape-rec? v) (pr-render-pairs buf
(map (fn [k] [k (shape-get v k nil)]) (shape-keys v)))
(core-sorted-map? v) (pr-render-pairs buf
(map (fn [e] [(vnth e 0) (vnth e 1)]) (sorted-entries-arr v)))
(core-sorted-set? v) (pr-render-seq buf (sorted-entries-arr v) "#{" "}")

1
src/jolt/evaluator.janet
View file

@ -106,6 +106,7 @@
[coll k default]
(cond
(jolt-transient? coll) (transient-lookup coll k default)
(shape-rec? coll) (shape-get coll k default)
# sorted colls are tables — without this arm they fell into the raw
# table-get branch and (:k (sorted-map ...)) was always nil (jolt-4vr spec)
(and (table? coll) (or (= :jolt/sorted-map (coll :jolt/type))

54
src/jolt/types.janet
View file

@ -590,3 +590,57 @@
(let [registry (get (ctx :env) :type-registry)
type-impls (get registry type-tag)]
(if (and type-impls (get type-impls protocol-name)) true false)))
# --- shape records (hidden classes, jolt-t34) -------------------------------
# A "shape record" is a cheap fixed-layout representation for a map literal
# whose keys are a known compile-time set (e.g. a vec3 {:r :g :b}). It is a
# Janet tuple [SHAPE v0 v1 ...] where SHAPE is an interned descriptor struct
# {:jolt/shape KEYS :idx {k->pos}}. Construction is a tuple (≈2x cheaper than a
# struct), const-keyword lookup compiles to an index, and general map ops below
# recognize it via shape-rec? and treat it as a map — so it is transparent
# wherever it flows. Created only by the backend when JOLT_SHAPE is on and the
# inference proves the shape; the runtime support here is always present so a
# shape value is handled correctly regardless.
(def- shape-cache @{}) # canonical-keys-tuple -> interned shape descriptor
# Canonical key order, OWNED BY THE RUNTIME (jolt-t34): every site that builds or
# reads a shape (shape-for, emit-map, emit-kw-lookup, build-map-literal) derives
# the layout from this one function, so they always agree regardless of what
# order the inference passed the keys in. Sorted by the keys' jdn print form —
# deterministic and total across keywords/strings/numbers/bools.
(defn shape-sort [ks]
(sort (array ;ks) (fn [a b] (< (string/format "%j" a) (string/format "%j" b)))))
(defn shape-for
"Interned shape descriptor for a key set. Keys are canonicalized internally,
so callers need not pre-sort and any permutation yields the same descriptor."
[keyv]
(def sk (tuple ;(shape-sort keyv)))
(or (get shape-cache sk)
(let [idx @{}]
(var i 0) (each k sk (put idx k i) (++ i))
(def desc (struct :jolt/shape sk :idx (table/to-struct idx)))
(put shape-cache sk desc)
desc)))
(defn shape-rec? [x]
(and (tuple? x) (> (length x) 0)
(struct? (in x 0)) (not (nil? (in (in x 0) :jolt/shape)))))
(defn shape-keys [rec] ((in rec 0) :jolt/shape))
(defn shape-get [rec k default]
(def pos (get ((in rec 0) :idx) k))
(if (nil? pos) default (in rec (+ pos 1))))
(defn shape-assoc [rec k v]
# assoc on a known key keeps the layout; a new key falls back to a struct
(def desc (in rec 0))
(def pos (get (desc :idx) k))
(if (nil? pos)
(let [t @{}] (each kk (desc :jolt/shape) (put t kk (shape-get rec kk nil))) (put t k v) (table/to-struct t))
(let [a (array ;rec)] (put a (+ pos 1) v) (tuple ;a))))
(defn shape-count [rec] (- (length rec) 1))
(defn shape-contains? [rec k] (not (nil? (get ((in rec 0) :idx) k))))
(defn shape-vals [rec] (tuple/slice rec 1))
# a struct snapshot of a shape-rec — the reusable bridge for ops that already
# handle structs (dissoc, vals, seq, equality, print, ...) without per-op code
(defn shape->struct [rec]
(def desc (in rec 0)) (def t @{})
(each kk (desc :jolt/shape) (put t kk (in rec (+ 1 (get (desc :idx) kk)))))
(table/to-struct t))

64
test/integration/shape-transparency-test.janet Normal file
View file

@ -0,0 +1,64 @@
# Shape-record transparency (jolt-t34 Round 1). A shape-rec is the compiler's
# cheap representation for a constant-key map literal — a Janet tuple
# [descriptor v0 v1 ...]. Every map operation must treat it EXACTLY like the
# equivalent struct map, so a shape value is transparent wherever it flows.
# These build shape-recs directly via the runtime (shape-for + tuple) and
# assert each op matches the struct map's behavior.
(use ../../src/jolt/types)
(use ../../src/jolt/core)
(var fails 0)
(defn check [label got want]
(if (deep= got want) (print " ok " label)
(do (++ fails) (printf " FAIL %s: got %j want %j" label got want))))
# {:a 1 :b 2} as a shape-rec and as a struct — they must be indistinguishable
(def SH (shape-for [:a :b]))
(def r (tuple SH 1 2))
(def s {:a 1 :b 2})
# --- access ------------------------------------------------------------------
(check "get hit" (core-get r :a nil) 1)
(check "get hit 2" (core-get r :b nil) 2)
(check "get miss" (core-get r :z :d) :d)
(check "count" (core-count r) 2)
(check "contains? hit" (core-contains? r :a) true)
(check "contains? miss" (core-contains? r :z) false)
(check "map?" (core-map? r) true)
# --- update (returns something that reads back correctly) --------------------
(check "assoc existing" (core-get (core-assoc r :a 9) :a nil) 9)
(check "assoc new key" (core-get (core-assoc r :c 3) :c nil) 3)
(check "assoc keeps others" (core-get (core-assoc r :c 3) :b nil) 2)
(check "dissoc" (core-get (core-dissoc r :a) :a :gone) :gone)
(check "dissoc keeps" (core-get (core-dissoc r :a) :b nil) 2)
# --- enumeration: entries via the central seq normalizer (what keys/vals/seq/
# reduce-kv in the overlay all flow through) — order-independent ---------------
(defn entryset [m] (sorted (map |(string/format "%j" $) (realize-for-iteration m))))
(check "entries match struct" (entryset r) (entryset s))
(check "count of seq" (length (core-seq r)) 2)
(check "first is a 2-entry" (length (core-first r)) 2)
# --- equality: a shape-rec equals the same struct map and vice versa ---------
(check "= shape vs struct" (jolt-equal? r s) true)
(check "= struct vs shape" (jolt-equal? s r) true)
(check "= shape vs shape" (jolt-equal? r (tuple (shape-for [:a :b]) 1 2)) true)
(check "not= diff value" (jolt-equal? r (tuple (shape-for [:a :b]) 1 9)) false)
(check "not= diff shape" (jolt-equal? r (tuple (shape-for [:a :b :c]) 1 2 3)) false)
(check "not= vs vector" (jolt-equal? r [1 2]) false)
# --- IFn: a map is callable as a key lookup ----------------------------------
(check "call as fn" (jolt-call r :a) 1)
(check "call as fn miss default" (jolt-call r :z :d) :d)
# --- nil/false values are present (shape-recs store them positionally) --------
(def rn (tuple (shape-for [:a :b]) nil false))
(check "nil value present" (core-contains? rn :a) true)
(check "nil value get" (core-get rn :a :d) nil)
(check "false value get" (core-get rn :b :d) false)
(check "count with nils" (core-count rn) 2)
(if (> fails 0)
(error (string "shape-transparency: " fails " failing check(s)"))
(print "\nShape transparency passed!"))