Collection fns: JVM-faithful return types + laziness (#219)

A type-aware audit (~190 collection expressions vs reference Clojure) found four
divergences the corpus missed — value-equality (= [0 1] '(0 1)) hides type and
laziness differences. Fixed, with type-predicate + over-infinite corpus rows that
pin them.

- partition-all [n coll] built vector chunks; JVM chunks are seqs. (The [n step
  coll] arity was already correct, as is the partition-all transducer, whose
  chunks are vectors in JVM too.) Now builds seq chunks.
- replace always returned a vector (mapv) and was eager; JVM is type-preserving —
  a vector maps to a vector, any other seqable to a lazy seq.
- sequence eagerly realized its source (into-xform), so (first (sequence (map inc)
  (range))) hung. Rewrote as a transformer iterator: pull one input at a time,
  buffer the step outputs, emit lazily, run the completion to flush a stateful
  xform. eduction builds on it (lazy, no longer an eager vector).
- mapcat and (apply concat coll-of-colls) hung over an infinite source because
  jolt-apply seq->lists the trailing arg and mapcat seq->lists the map result.
  Added lazy-concat-seq (lazily flatten a seq of colls); mapcat uses it directly,
  and apply special-cases concat (its result is lazy) to route through it.

Docs: a cross-cutting return-type + laziness contract in docs/spec/09-core-library;
SPEC.md notes that = masks type/laziness so they need predicate / over-infinite
rows. EBNF is reader syntax only — unaffected.

Seed change (partition-all/replace/eduction are clojure.core overlay) -> re-mint;
selfhost holds. make test + shakesmoke + buildsmoke green, 0 new divergences.

Co-authored-by: Yogthos <yogthos@gmail.com>

docs/spec/09-core-library.md

@@ -10,6 +10,40 @@ them (e.g. vector `nth` is "effectively constant time" — SHOULD-level).
 
 ---
 
+## Collection return types & laziness (cross-cutting)
+
+Two contracts hold across the sequence library and are not restated per entry.
+
+**Return-type fidelity.** A function returns the same *kind* of collection the
+reference does — value equality is not enough, since `(= [0 1] '(0 1))`.
+
+- Sequence transformations return **seqs** (lazy unless noted): `map`, `filter`,
+  `remove`, `keep`, `mapcat`, `take`/`drop` and their `-while` forms, `partition`,
+  `partition-all`, `partition-by`, `interpose`, `dedupe`, `distinct`, `concat`,
+  `reductions`, `cons`, `rest`, `sequence`. The *elements* of `partition` /
+  `partition-all` / `partition-by` are themselves seqs, not vectors.
+- The vector variants return **vectors**: `mapv`, `filterv`, `vec`, `subvec`,
+  `partitionv`, `partitionv-all`, `splitv-at`. `split-at` / `split-with` return a
+  2-vector `[take drop]`. A transducer applied eagerly (`into []`, the
+  `partition-all` transducer's chunks) yields vectors.
+- Type-preserving functions return the input's type: `replace` over a vector is a
+  vector, over any other seqable a (lazy) seq; `empty`/`into (empty coll)` keep the
+  collection kind; `set`/`into #{}` return sets; `into {}`/`select-keys`/`zipmap`/
+  `frequencies`/`group-by`/`merge` return maps (`group-by` values are vectors).
+
+**Laziness.** The lazy sequence functions — including `sequence`, `eduction`, and
+`mapcat` — MUST consume their source incrementally and so terminate on an infinite
+or unbounded source when only a prefix is demanded: `(first (sequence (map inc)
+(range)))` and `(take n (mapcat f (range)))` return without realizing the whole
+source. `(apply concat coll-of-colls)` is likewise lazy in its argument seq. The
+eager consumers (`reduce`, `into`, `count`, `vec`, `doall`) realize the demanded
+portion fully.
+
+These are exercised by the `seq / lazy over infinite` and the per-fn type-predicate
+rows in the conformance corpus.
+
+---
+
 ### first — since 1.0
 
 ```

host/chez/natives-seq.ss

@@ -101,7 +101,9 @@
 (define (jolt-mapcat f . colls)
   (if (null? colls)
       (td-mapcat f)
-      (apply jolt-concat (seq->list (apply jolt-map f colls)))))
+      ;; lazily concat the per-element results — no seq->list, so mapcat over an
+      ;; infinite source stays lazy.
+      (lazy-concat-seq (apply jolt-map f colls))))
 
 ;; take-while / drop-while: 1-arg -> transducer; 2-arg -> a seq over the coll.
 (define (take-while-seq pred s)

host/chez/natives-transduce.ss

@@ -38,13 +38,54 @@
             (res (reduce-seq xf init (jolt-seq coll))))
        (jolt-invoke xf res)))))
 
-;; (sequence coll) -> a seq; (sequence xform coll) -> coll transformed by xform.
+;; (sequence coll) -> a seq; (sequence xform coll) -> a LAZY seq of coll transformed
-;; Materialized eagerly through into-xform then seq'd (corpus inputs are finite; a
+;; by xform. A transformer iterator (mirrors clojure.core's TransformerIterator):
-;; fully-lazy pull is future work). Honors reduced via into-xform/reduce-seq.
+;; pull one input at a time through (xform rf), where rf buffers each emitted value;
+;; emit the buffer lazily, pulling more input only when it drains. So an infinite or
+;; expensive source is consumed incrementally — (first (sequence (map inc) (range)))
+;; returns at once. Honors `reduced` (stop pulling) and runs the 1-arg completion to
+;; flush a stateful xform (partition-all / dedupe / a trailing partition).
+(define (sequence-xf xform coll)
+  (let* ((buf (box '()))                  ; emitted values for the current step, reversed
+         (rf (case-lambda
+               (() jolt-nil)
+               ((acc) acc)
+               ((acc x) (set-box! buf (cons x (unbox buf))) acc)))
+         (xrf (jolt-invoke xform rf)))
+    ;; advance the source until buf holds output or the input is drained+completed.
+    (define (fill src acc completed)
+      (let loop ((src src) (acc acc) (completed completed))
+        (cond
+          ((pair? (unbox buf)) (values src acc completed))
+          (completed (values src acc #t))
+          ((jolt-reduced? acc)
+           (jolt-invoke xrf (jolt-reduced-val acc))      ; completion may flush
+           (loop src (jolt-reduced-val acc) #t))
+          (else
+           (let ((s (jolt-seq src)))
+             (if (jolt-nil? s)
+                 (begin (jolt-invoke xrf acc) (loop src acc #t))   ; complete -> flush
+                 (loop (seq-more s) (jolt-invoke xrf acc (seq-first s)) completed)))))))
+    ;; Resolve the next chunk now (one fill pulls just enough input to emit or to
+    ;; exhaust), so the result is a real cseq | empty — `empty` is jolt-empty-list
+    ;; at the top (so an empty result still prints "()") and jolt-nil inside a tail
+    ;; (the cseq terminator). The TAILS stay lazy, so an infinite source is fine.
+    (define (step src acc completed empty)
+      (let-values (((src2 acc2 comp2) (fill src acc completed)))
+        (let ((out (reverse (unbox buf))))
+          (set-box! buf '())
+          (if (null? out)
+              empty
+              (let build ((o out))
+                (if (null? (cdr o))
+                    (cseq-lazy (car o) (lambda () (step src2 acc2 comp2 jolt-nil)))
+                    (cseq-lazy (car o) (lambda () (build (cdr o))))))))))
+    (step coll jolt-nil #f jolt-empty-list)))
+
 (define jolt-sequence
   (case-lambda
     ((coll) (jolt-seq coll))
-    ((xform coll) (jolt-seq (into-xform (jolt-vector) xform coll)))))
+    ((xform coll) (sequence-xf xform coll))))
 
 (def-var! "clojure.core" "transduce" jolt-transduce)
 (def-var! "clojure.core" "sequence" jolt-sequence)

host/chez/seq.ss

@@ -269,10 +269,25 @@
         ((null? (cdr colls)) (jolt-seq (car colls)))
         (else (concat2 (jolt-seq (car colls)) (lambda () (apply jolt-concat (cdr colls)))))))
 
-;; (apply f a b ... coll): spread the trailing seqable into the call.
+;; Lazily concatenate a (possibly infinite) SEQ of colls — what (apply concat ss)
+;; means, but without realizing ss. Pulls one coll at a time, concatenating it with
+;; a lazy tail, so mapcat / (apply concat …) over an infinite source stays lazy.
+(define (lazy-concat-seq ss)
+  (let ((s (jolt-seq ss)))
+    (if (jolt-nil? s)
+        jolt-empty-list
+        (jolt-concat (seq-first s)
+                     (jolt-make-lazy-seq (lambda () (lazy-concat-seq (seq-more s))))))))
+
+;; (apply f a b ... coll): spread the trailing seqable into the call. concat is
+;; special-cased: it produces a LAZY result, so spreading an infinite tail through
+;; a Scheme variadic (which must realize it) would hang — route to lazy-concat-seq,
+;; prepending any fixed leading colls.
 (define (jolt-apply f . args)
-  (let* ((r (reverse args)) (spread (seq->list (jolt-seq (car r)))) (fixed (reverse (cdr r))))
+  (let* ((r (reverse args)) (tail (car r)) (fixed (reverse (cdr r))))
-    (apply jolt-invoke f (append fixed spread))))
+    (if (eq? f jolt-concat)
+        (lazy-concat-seq (fold-right jolt-cons (jolt-seq tail) fixed))
+        (apply jolt-invoke f (append fixed (seq->list (jolt-seq tail)))))))
 
 ;; ============================================================================
 ;; numeric predicates / identity — usable in fn AND value position (map/filter).

jolt-core/clojure/core/20-coll.clj

@@ -210,7 +210,12 @@
          true))
      false)))
 
-(defn replace [smap coll] (mapv (fn [x] (get smap x x)) coll))
+;; A vector input maps to a vector (eager); any other coll to a lazy seq — JVM
+;; replace is type-preserving, not vector-always.
+(defn replace [smap coll]
+  (if (vector? coll)
+    (mapv (fn [x] (get smap x x)) coll)
+    (map (fn [x] (get smap x x)) coll)))
 
 (defn nthnext [coll n]
   (loop [n n xs (seq coll)]

jolt-core/clojure/core/22-coll.clj

@@ -271,14 +271,17 @@
 
 ;; eduction is EAGER on jolt (documented divergence): the composed
 ;; xforms applied to coll, realized into a vector.
+;; A lazy application of the composed xforms to coll (sequence is lazy now), so an
+;; infinite or expensive source isn't realized up front. Not a re-iterable Eduction
+;; object, but reduce / into / seq / first over it all work.
 (defn eduction [& args]
   (let [coll (last args)
         xforms (butlast args)]
     (if xforms
-      (into [] (apply comp xforms) coll)
+      (sequence (apply comp xforms) coll)
-      (into [] coll))))
+      (sequence coll))))
 
-(defn ->Eduction [xform coll] (into [] xform coll))
+(defn ->Eduction [xform coll] (sequence xform coll))
 
 ;; --- JVM-shape stubs and trivial shells --------------------------------------
 ;; Pure compositions or documented jolt stubs; the host keeps nothing.

jolt-core/clojure/core/40-lazy.clj

@@ -125,10 +125,13 @@
    (letfn [(go [s]
              (lazy-seq
                (when (seq s)
-                 (loop [i 0 chunk [] cur s]
+                 ;; realize exactly n via first/rest (minimal realization), but emit
+                 ;; the chunk as a SEQ — JVM partition-all chunks are seqs, not
+                 ;; vectors (partitionv-all is the vector variant).
+                 (loop [i 0 acc () cur s]
                    (if (and (< i n) (seq cur))
-                     (recur (inc i) (conj chunk (first cur)) (rest cur))
+                     (recur (inc i) (cons (first cur) acc) (rest cur))
-                     (cons chunk (go cur)))))))]
+                     (cons (reverse acc) (go cur)))))))]
      (go coll)))
   ([n step coll]
    (letfn [(go [s]

test/chez/corpus.edn

@@ -1899,6 +1899,20 @@
   {:suite "seq / map filter reduce" :label "mapcat three colls" :expected "[1 2 3]" :actual "(mapcat vector [1] [2] [3])"}
   {:suite "seq / map filter reduce" :label "mapcat empty coll" :expected "[]" :actual "(mapcat vector [] [1 2] [3 4])"}
   {:suite "seq / map filter reduce" :label "mapcat seqs" :expected "[1 2 3 4]" :actual "(mapcat identity [[1 2] [3 4]])"}
+  {:suite "seq / lazy over infinite" :label "mapcat is lazy over an infinite source" :expected "[1 2 1 2]" :actual "(take 4 (mapcat identity (repeat [1 2])))"}
+  {:suite "seq / lazy over infinite" :label "mapcat single coll lazy" :expected "[0 1 2]" :actual "(take 3 (mapcat vector (range)))"}
+  {:suite "seq / lazy over infinite" :label "apply concat lazy over infinite" :expected "[1 1 1]" :actual "(take 3 (apply concat (repeat [1])))"}
+  {:suite "seq / lazy over infinite" :label "apply concat fixed + infinite tail" :expected "[9 0 0 0 0]" :actual "(take 5 (apply concat [9] (repeat [0])))"}
+  {:suite "seq / lazy over infinite" :label "sequence is lazy over an infinite source" :expected "[1 2 3]" :actual "(take 3 (sequence (map inc) (range)))"}
+  {:suite "seq / lazy over infinite" :label "sequence first of infinite" :expected "2" :actual "(first (sequence (map inc) (range 1 100000000000)))"}
+  {:suite "seq / lazy over infinite" :label "sequence stateful xform lazy" :expected "[[0 1] [2 3]]" :actual "(take 2 (sequence (partition-all 2) (range)))"}
+  {:suite "seq / lazy over infinite" :label "sequence value + completion flush" :expected "[[0 1] [2 3] [4]]" :actual "(sequence (partition-all 2) (range 5))"}
+  {:suite "seq / lazy over infinite" :label "sequence empty stays a seq" :expected "true" :actual "(seq? (sequence (filter even?) [1 3 5]))"}
+  {:suite "seq / lazy over infinite" :label "sequence empty prints as empty" :expected "()" :actual "(sequence (filter even?) [1 3 5])"}
+  {:suite "seq / lazy over infinite" :label "eduction is lazy over infinite" :expected "1" :actual "(first (eduction (map inc) (range)))"}
+  {:suite "seq / lazy over infinite" :label "eduction reduces" :expected "9" :actual "(reduce + (eduction (filter odd?) [1 2 3 4 5]))"}
+  {:suite "seq / lazy over infinite" :label "eduction into" :expected "[2 3 4]" :actual "(into [] (eduction (map inc) [1 2 3]))"}
+  {:suite "seq / lazy over infinite" :label "eduction multiple xforms compose left-to-right" :expected "[2 4]" :actual "(into [] (eduction (filter odd?) (map inc) [1 2 3 4]))"}
   {:suite "seq / map filter reduce" :label "keep" :expected "[1 3]" :actual "(keep (fn [x] (if (odd? x) x nil)) [1 2 3 4])"}
   {:suite "seq / map filter reduce" :label "some truthy" :expected "true" :actual "(some even? [1 2 3])"}
   {:suite "seq / map filter reduce" :label "some nil" :expected "nil" :actual "(some even? [1 3 5])"}
@@ -1914,6 +1928,11 @@
   {:suite "seq / take drop slice" :label "take-nth" :expected "[1 3 5]" :actual "(take-nth 2 [1 2 3 4 5])"}
   {:suite "seq / take drop slice" :label "partition" :expected "[[1 2] [3 4]]" :actual "(partition 2 [1 2 3 4 5])"}
   {:suite "seq / take drop slice" :label "partition-all" :expected "[[1 2] [3]]" :actual "(partition-all 2 [1 2 3])"}
+  {:suite "seq / take drop slice" :label "partition elems are seqs" :expected "true" :actual "(every? seq? (partition 2 [1 2 3 4]))"}
+  {:suite "seq / take drop slice" :label "partition-all elems are seqs not vectors" :expected "true" :actual "(every? seq? (partition-all 2 (range 5)))"}
+  {:suite "seq / take drop slice" :label "partition-all elem is not a vector" :expected "false" :actual "(vector? (first (partition-all 2 [1 2 3])))"}
+  {:suite "seq / take drop slice" :label "partition-all [n step coll] elems are seqs" :expected "true" :actual "(every? seq? (partition-all 2 2 [1 2 3 4]))"}
+  {:suite "seq / take drop slice" :label "partition-by elems are seqs" :expected "true" :actual "(every? seq? (partition-by odd? [1 3 2 4]))"}
   {:suite "seq / take drop slice" :label "split-at" :expected "[[1 2] [3 4]]" :actual "(split-at 2 [1 2 3 4])"}
   {:suite "seq / transform" :label "reverse" :expected "[3 2 1]" :actual "(reverse [1 2 3])"}
   {:suite "seq / transform" :label "sort" :expected "[1 2 3]" :actual "(sort [3 1 2])"}
@@ -2026,6 +2045,9 @@
   {:suite "seq / overlay-migrated fns" :label "distinct? single" :expected "true" :actual "(distinct? 5)"}
   {:suite "seq / overlay-migrated fns" :label "replace maps elements" :expected "[:a 2 :c 2]" :actual "(replace {1 :a 3 :c} [1 2 3 2])"}
   {:suite "seq / overlay-migrated fns" :label "replace preserves nil val" :expected "[1 nil 3]" :actual "(replace {2 nil} [1 2 3])"}
+  {:suite "seq / overlay-migrated fns" :label "replace on a vector stays a vector" :expected "true" :actual "(vector? (replace {1 :a} [1 2 1]))"}
+  {:suite "seq / overlay-migrated fns" :label "replace on a seq returns a seq" :expected "true" :actual "(seq? (replace {1 :a} (list 1 2 1)))"}
+  {:suite "seq / overlay-migrated fns" :label "replace on a seq is lazy" :expected "(0 :a 2)" :actual "(take 3 (replace {1 :a} (range)))"}
   {:suite "seq / overlay-migrated fns" :label "take-last" :expected "[3 4]" :actual "(take-last 2 [1 2 3 4])"}
   {:suite "seq / overlay-migrated fns" :label "take-last empty -> nil" :expected "nil" :actual "(take-last 2 [])"}
   {:suite "seq / overlay-migrated fns" :label "take-last n>len" :expected "[1 2]" :actual "(take-last 9 [1 2])"}

test/conformance/SPEC.md

@@ -35,6 +35,12 @@ the canonical, frozen contract**: it is what every runtime consumes, what
   disambiguates duplicate labels with ` (N)`).
 - Comparison is **value-equality** (`=`), never string/printed-form — so map/set
   iteration order never matters.
+- Because comparison is `=`, a **type** or **laziness** difference is invisible to a
+  plain value row: `(= [0 1] '(0 1))` is true, so a fn returning a vector where
+  Clojure returns a seq still passes. Pin those explicitly — container/element type
+  with a predicate row (`(seq? …)`, `(vector? …)`, `(every? seq? …)`), and laziness
+  with a `(take n (… (range)))` row over an infinite source (it hangs, not just
+  diverges, if the fn isn't lazy). The `seq / lazy over infinite` suite does both.
 - `:expected :throws` asserts evaluating `:actual` raises.
 
 ## The oracle: reference JVM Clojure