simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Chunk range/map/filter to match JVM Clojure

Browse source 
range, map, and filter were fully element-by-element lazy, so
(map f (range 1 50)) realized one element per first/nth where JVM
Clojure realizes a whole 32-element chunk. range is a chunked
LongRange on the JVM and map/filter are chunk-preserving, so the
observable side-effect timing differed.

Following clojure.lang.LongRange, ChunkedCons, ChunkBuffer and
core.clj, this adds a crest field to the cseq record and a
cseq-chunked constructor modeling ChunkedCons (a standalone chunk
pvec, an offset, and the after-chunk seq). The chunk accessors move
to seq.ss next to the representation they read. map/filter/remove
take a chunked branch when the source is chunked, realizing the whole
chunk and chunk-cons'ing it onto a lazy rest, so their output is
itself chunked and chained transforms each batch by 32. Bounded range
is now an eager chunked seq, and the reduce fast path flows through a
ChunkedCons rest. The chunk-buffer/chunk/chunk-cons builder API in
natives-array.ss now produces a real ChunkedCons.

Single-arg (range), multi-coll map, and plain lazy seqs stay
element-by-element, like the JVM.

Adds a lazy / chunking suite to the corpus that observes realization
timing via an atom counter: first over a chunked map realizes 32,
crossing a chunk boundary realizes 49, chained maps batch [32 32],
filter applies the predicate to the whole first block, and a plain
lazy seq still realizes one element at a time. Two cases that
documented the old over-laziness now assert the JVM value of 32 and
were dropped from the allowlist. certify against JVM Clojure 1.12.3
reports 0 new and 0 stale divergences.
This commit is contained in:
Yogthos 2026-06-29 22:02:06 -04:00
parent c0a0ec98ee
commit bd33d605ef
5 changed files with 170 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

21
test/chez/corpus.edn
View file

@ -2728,7 +2728,7 @@
{:suite "untested / unchecked-* are plain ops" :label "unchecked-byte" :expected "65" :actual "(unchecked-byte 65)"}
{:suite "untested / unchecked-* are plain ops" :label "unchecked-char" :expected "\\a" :actual "(unchecked-char 97)"}
{:suite "untested / unchecked-* are plain ops" :label "unchecked-short" :expected "5" :actual "(unchecked-short 5)"}
{:suite "untested / chunk family (eager equivalents) + cat" :label "chunk round-trip" :expected "1" :actual "(let [cb (chunk-buffer 4)] (chunk-append cb 1) (chunk-first (chunk-cons (chunk cb) nil)))"}
{:suite "untested / chunk family (eager equivalents) + cat" :label "chunk round-trip" :expected "[1]" :actual "(let [cb (chunk-buffer 4)] (chunk-append cb 1) (chunk-first (chunk-cons (chunk cb) nil)))"}
{:suite "untested / chunk family (eager equivalents) + cat" :label "cat transducer" :expected "[1 2 3]" :actual "(into [] cat [[1] [2 3]])"}
{:suite "untested / chunk family (eager equivalents) + cat" :label "ensure-reduced wraps" :expected "true" :actual "(reduced? (ensure-reduced 5))"}
{:suite "untested / chunk family (eager equivalents) + cat" :label "ensure-reduced keeps reduced" :expected "true" :actual "(reduced? (ensure-reduced (reduced 5)))"}
@ -2755,11 +2755,14 @@
{:suite "seq-type-model / specialized seq classes collapse" :label "sort" :expected "clojure.lang.PersistentList" :actual "(class (sort [3 1 2]))"}
{:suite "seq-type-model / specialized seq classes collapse" :label "subvec" :expected "clojure.lang.PersistentVector" :actual "(class (subvec [1 2 3] 1))"}
{:suite "seq-type-model / specialized seq classes collapse" :label "drop over a vector" :expected "clojure.lang.LazySeq" :actual "(class (drop 1 [1 2 3]))"}
;; --- chunking-model divergences (allowlisted): jolt is unchunked, so forcing
;; one element realizes 1 (JVM realizes a 32-element chunk), and mapcat/dedupe
;; realize 0 at construction (JVM forces the first chunk). jolt-mm6v.
{:suite "chunking-model / unchunked realization granularity" :label "first over a vector realizes one, not a chunk" :expected "1" :actual "(let [a (atom 0)] (first (map (fn [x] (swap! a inc) x) (vec (range 100)))) @a)"}
{:suite "chunking-model / unchunked realization granularity" :label "nth 0 over a vector realizes one" :expected "1" :actual "(let [a (atom 0)] (nth (map (fn [x] (swap! a inc) x) (vec (range 100))) 0) @a)"}
;; --- chunking-model realization granularity (JVM-matching): a vector's seq is
;; chunked, so forcing one element of (map f a-vector) realizes the whole first
;; 32-block, like the JVM.
{:suite "chunking-model / realization granularity" :label "first over a chunked vector realizes the whole 32-block" :expected "32" :actual "(let [a (atom 0)] (first (map (fn [x] (swap! a inc) x) (vec (range 100)))) @a)"}
{:suite "chunking-model / realization granularity" :label "nth 0 over a chunked vector realizes the whole 32-block" :expected "32" :actual "(let [a (atom 0)] (nth (map (fn [x] (swap! a inc) x) (vec (range 100))) 0) @a)"}
;; mapcat/dedupe stay lazier than the JVM at construction: jolt's (apply concat …)
;; and sequence transformer don't force the first chunk just to build the seq
;; (the JVM forces 5 here). Allowlisted in known-divergences.edn (jolt-mm6v).
{:suite "chunking-model / unchunked realization granularity" :label "mapcat is fully lazy at construction" :expected "0" :actual "(let [a (atom 0)] (mapcat (fn [x] (swap! a inc) [x]) (range 5)) @a)"}
{:suite "chunking-model / unchunked realization granularity" :label "dedupe is fully lazy at construction" :expected "0" :actual "(let [a (atom 0)] (dedupe (map (fn [x] (swap! a inc) x) (range 5))) @a)"}
;; --- integer-box-model: narrow int values behave as integers (certified) ---
@ -3483,4 +3486,10 @@
{:suite "host-interop / a fn reports its munged class" :label "(class a-def'd-fn) is ns$munged-name, with the IFn hierarchy as ancestors" :expected "[\"clojure.core$odd_QMARK_\" true]" :actual "[(.getName (class odd?)) (boolean (some #{java.lang.Runnable} (ancestors (class odd?))))]"}
{:suite "host-interop / MultiFn methods" :label ".getMethod / .dispatchFn on a multimethod" :expected "[true true true]" :actual "(do (defmulti mmc identity) (defmethod mmc :a [_] 1) [(some? (.getMethod mmc :a)) (nil? (.getMethod mmc :z)) (ifn? (.dispatchFn mmc))])"}
{:suite "reader / namespaced map literal" :label "#:ns{...} qualifies bare keys; :_/x stays unqualified" :expected "[:search 2 nil]" :actual "[(:event/type #:event{:type :search :id 5}) (count #:x{:a 1 :b 2}) (:k #:_{:k 9})]"}
{:suite "lazy / chunking" :label "(seq (range 1 50)) is a chunked-seq" :expected "true" :actual "(chunked-seq? (seq (range 1 50)))"}
{:suite "lazy / chunking" :label "map over a chunked range realizes a whole 32-block on first" :expected "32" :actual "(let [c (atom 0) s (map (fn [x] (swap! c inc) x) (range 1 50))] (first s) @c)"}
{:suite "lazy / chunking" :label "crossing a chunk boundary realizes the next 32-block" :expected "49" :actual "(let [c (atom 0) s (map (fn [x] (swap! c inc) x) (range 1 50))] (nth s 32) @c)"}
{:suite "lazy / chunking" :label "chained maps each batch by 32 (result is itself chunked)" :expected "[32 32]" :actual "(let [fc (atom 0) gc (atom 0) s (map (fn [x] (swap! gc inc) x) (map (fn [x] (swap! fc inc) x) (range 1 50)))] (first s) [@fc @gc])"}
{:suite "lazy / chunking" :label "filter over a chunked range applies pred to the whole first block" :expected "32" :actual "(let [c (atom 0) s (filter (fn [x] (swap! c inc) (even? x)) (range 1 50))] (first s) @c)"}
{:suite "lazy / chunking" :label "a plain lazy seq (not chunked) realizes one element at a time" :expected "1" :actual "(let [c (atom 0) lz (fn lz [n] (lazy-seq (cons n (lz (inc n))))) s (map (fn [x] (swap! c inc) x) (lz 0))] (first s) @c)"}
]

4
test/conformance/known-divergences.edn
View file

@ -16,7 +16,7 @@
:seq-type-model
"jolt models every seq as PersistentList (eager) or LazySeq (deferred); JVM reifies a specialized class per producer (Cons/Iterate/LongRange/Repeat/Cycle/ChunkedSeq/StringSeq/KeySeq/RSeq/ArraySeq/SubVector). Values and laziness are correct, only (class …) differs. jolt-aei7",
:chunking-model
"jolt seqs are unchunked: forcing one element realizes one (JVM realizes a ~32-element chunk), and mapcat/dedupe realize 0 at construction where JVM forces the first chunk — jolt is a finer-grained lazy superset. jolt-mm6v",
"jolt now chunks range/vector seqs through map/filter like the JVM (forcing one element realizes the whole ~32-element chunk). What remains finer-grained: mapcat/dedupe realize 0 at construction where the JVM's (apply concat …) / sequence transformer force the first chunk just to build the seq. jolt-mm6v",
:integer-box-model
"jolt unifies every integer as one exact-integer type. A Chez fixnum is an immediate identical to the plain integer (no distinct identity to tag, no metadata on numbers), so (byte/short/int n) report Long, not Byte/Short/Integer. Value/arithmetic/equality are correct; a faithful narrow box would crash raw compiled (+ …) or de-optimize all arithmetic. jolt-k9sw"},
:entries
@ -53,8 +53,6 @@
{:suite "seq-type-model / specialized seq classes collapse", :label "sort", :category :seq-type-model}
{:suite "seq-type-model / specialized seq classes collapse", :label "subvec", :category :seq-type-model}
{:suite "seq-type-model / specialized seq classes collapse", :label "drop over a vector", :category :seq-type-model}
{:suite "chunking-model / unchunked realization granularity", :label "first over a vector realizes one, not a chunk", :category :chunking-model}
{:suite "chunking-model / unchunked realization granularity", :label "nth 0 over a vector realizes one", :category :chunking-model}
{:suite "chunking-model / unchunked realization granularity", :label "mapcat is fully lazy at construction", :category :chunking-model}
{:suite "chunking-model / unchunked realization granularity", :label "dedupe is fully lazy at construction", :category :chunking-model}
{:suite "integer-box-model / narrow int class collapses to Long", :label "byte class", :category :integer-box-model}