simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chunk-first: pull the trie leaf instead of flattening the whole vector

Browse source 
A pvec is a 32-way trie, but na-chunk-first built each block by calling
pvec-v on the full backing vector — materializing all n elements to a
flat Scheme vector — then copying the 32-wide window out of it. That made
chunk-first O(n), so walking a vector chunk-by-chunk (Clojure's real
chunked map/filter fast path) was O(n^2): a ported chunked map over 500K
elements took 39s, superlinear to ~700s at 2M.

na-chunk-size equals pv-width and blocks are 32-aligned, so a block is
exactly one trie leaf — pv-chunk-for hands it back in O(log n). Copy that
leaf directly; fall back to per-index reads for the rare window that
crosses a leaf boundary. Chunked map is now linear, ~133x faster at 500K
(293ms) and within ~2.3x of the native seq loop, which makes a
clojure-in-clojure seq tier viable.

Corpus rows pin chunk-first window contents + chunk-rest boundaries
against JVM; fixed a stale 'always false' chunked-seq? label.
This commit is contained in:
Yogthos 2026-06-28 01:56:26 -04:00
parent 6940b2c7f5
commit 6d441e2d00
2 changed files with 22 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

17
host/chez/java/natives-array.ss
View file

@ -109,9 +109,24 @@
(let* ((v (cseq-cvec s)) (i (cseq-ci s)))
(cons v (fxmin (fx+ i na-chunk-size) (pvec-count v))))))
(define (na-chunked-seq? x) (and (na-vblock x) #t))
;; Copy the block [i, end) straight out of the pvec trie's 32-element leaf node
;; (pv-chunk-for is O(log n)). na-chunk-size == pv-width and blocks are 32-aligned,
;; so a block is exactly one leaf; the rare non-aligned window crossing a leaf
;; boundary falls back to per-index reads. Flattening the whole backing vector
;; per block (pvec-v) made chunk-first O(n), so walking a vector chunk-by-chunk
;; was O(n^2).
(define (na-chunk-first s)
(let ((vb (na-vblock s)))
(if vb (make-pvec (vec-copy-range (pvec-v (car vb)) (cseq-ci s) (cdr vb)))
(if vb
(let* ((pv (car vb)) (i (cseq-ci s)) (end (cdr vb)) (len (fx- end i))
(node (pv-chunk-for pv i)) (off (fxand i pv-mask)))
(if (fx<=? (fx+ off len) (vector-length node))
(make-pvec (vec-copy-range node off (fx+ off len)))
(let ((out (make-vector len)))
(let loop ((j 0))
(if (fx<? j len)
(begin (vector-set! out j (pvec-nth-d pv (fx+ i j) jolt-nil)) (loop (fx+ j 1)))
(make-pvec out))))))
(jolt-first s)))) ; eager-buffer fallback
(define (na-chunk-rest s)
(let ((vb (na-vblock s)))

7
test/chez/corpus.edn
View file

@ -1640,7 +1640,7 @@
{:suite "predicates / tagged-value" :label "tagged-literal? yes" :expected "true" :actual "(tagged-literal? (tagged-literal (quote inst) \"2020\"))"}
{:suite "predicates / tagged-value" :label "tagged-literal? no" :expected "false" :actual "(tagged-literal? 1)"}
{:suite "predicates / tagged-value" :label "reader-conditional? no" :expected "false" :actual "(reader-conditional? 1)"}
{:suite "predicates / tagged-value" :label "chunked-seq? always false" :expected "true" :actual "(chunked-seq? (seq [1 2 3]))"}
{:suite "predicates / tagged-value" :label "chunked-seq? true for a vector seq" :expected "true" :actual "(chunked-seq? (seq [1 2 3]))"}
{:suite "predicates / equality & identity" :label "= same" :expected "true" :actual "(= 1 1)"}
{:suite "predicates / equality & identity" :label "= vectors" :expected "true" :actual "(= [1 2] [1 2])"}
{:suite "predicates / equality & identity" :label "= vector & list" :expected "true" :actual "(= [1 2] (list 1 2))"}
@ -2709,6 +2709,11 @@
{:suite "untested / chunk family (eager equivalents) + cat" :label "halt-when" :expected "4" :actual "(transduce (halt-when even?) conj [] [1 3 4 5])"}
{:suite "untested / chunk family (eager equivalents) + cat" :label "chunk-next exhausted" :expected "nil" :actual "(let [cb (chunk-buffer 2)] (chunk-append cb 1) (chunk-next (chunk-cons (chunk cb) nil)))"}
{:suite "untested / chunk family (eager equivalents) + cat" :label "chunk-rest seqable" :expected "[]" :actual "(let [cb (chunk-buffer 2)] (chunk-append cb 1) (vec (chunk-rest (chunk-cons (chunk cb) nil))))"}
{:suite "chunked-seq / over a vector" :label "chunk-first is a 32-element block" :expected "32" :actual "(count (chunk-first (seq (vec (range 100)))))"}
{:suite "chunked-seq / over a vector" :label "chunked-seq? true for vector seq" :expected "true" :actual "(chunked-seq? (seq (vec (range 100))))"}
{:suite "chunked-seq / over a vector" :label "chunked-seq? false for a list seq" :expected "false" :actual "(chunked-seq? (seq (list 1 2 3)))"}
{:suite "chunked-seq / over a vector" :label "chunk-first contents + chunk-rest boundary" :expected "[32 0 31 32]" :actual "(let [s (seq (vec (range 50))) c (chunk-first s)] [(count c) (nth c 0) (nth c 31) (first (chunk-rest s))])"}
{:suite "chunked-seq / over a vector" :label "chunk-first window past the first block" :expected "[32 33 34 35 36 37 38 39]" :actual "(vec (chunk-first (chunk-rest (seq (vec (range 40))))))"}
{:suite "untested / JVM-shape stubs (documented jolt behavior)" :label "class number" :expected "java.lang.Long" :actual "(class 1)"}
{:suite "untested / JVM-shape stubs (documented jolt behavior)" :label "class string" :expected "java.lang.String" :actual "(class \"s\")"}
{:suite "untested / JVM-shape stubs (documented jolt behavior)" :label "class keyword" :expected "clojure.lang.Keyword" :actual "(class :k)"}