jolt/phase-5.md

# Phase 5 — True Laziness (jolt-c09)

Final phase of the `jolt-1j0` clojure.core migration epic. Make jolt's sequence
generators and transformers genuinely lazy, so infinite seqs and lazy
compositions work and stop hanging the evaluator. This is the deepest and
riskiest phase — sub-stage it and gate every step.

> Issue: `bd show jolt-c09`. Depends on Phase 4 (`jolt-ldf`, done). Blocks nothing
> — it's the last phase.

---

## 1. Current state (what already works, what doesn't)

**The LazySeq machinery exists and is sound.** (`src/jolt/phm.janet`)
- A LazySeq is `@{:jolt/type :jolt/lazy-seq :fn thunk :realized false :val nil}`.
- A thunk returns `nil` (empty) or a cons cell `[first-val rest-thunk]`.
- `realize-ls` forces one cell (memoized via `:realized`), with a `:jolt/pending`
  sentinel that makes **self-referential** seqs work (`(def ones (lazy-seq (cons 1 ones)))`).
- `ls-first` / `ls-rest` / `ls-seq` / `ls-count` walk it. `lazy-seq?` detects it.

**Already lazy (keep):**
- Infinite generators: `(range)`, `(repeat x)`, `(iterate f x)`, `(cycle ...)`,
  `repeatedly` return LazySeq. Bounded forms (`(range n)`, `(repeat n x)`) are
  eager tuples/arrays — correct, they're finite.
- `map`/`filter` are **hybrid**: lazy when the input is a LazySeq, eager (and
  representation-preserving) when the input is a concrete collection.
- `take`/`drop`/`take-while` pull lazily from a LazySeq input but **return an eager
  array** (fine for bounded `take`, wrong for the others on infinite tails).
- Conformance already covers the working cases (self-ref fib, `iterate`, `count`
  of `take`, `filter`/`take-while`/`remove` over `(range)`): see
  `test/integration/conformance-test.janet` lines ~21–143.

**The gaps (what hangs):**
1. **Eager transformers that force their input** even when it's infinite. Confirmed
   callers of `realize-for-iteration` in their bodies: `remove`, `interpose`,
   `distinct`, `take-nth`, `map-indexed`, `keep-indexed`, `partition-all`,
   `partition-by`, `drop-while`. Plus `partition`, `interleave`, `concat`,
   `dedupe`, `flatten`, `tree-seq`, `mapcat`, `keep`, `sequence` need an
   infinite-input audit.
2. **`map`/`filter` over a *concrete vector* return an eager array**, not a lazy
   seq. Clojure returns a lazy seq. This is a **representation decision** (§3 Step 6).
3. **`realize-for-iteration` is the universal forcing point** (57 call sites). Many
   are legitimate realization boundaries (`count`, `into`, `reduce`, `vec`, `pr`),
   but any transformer that calls it on a lazy input loses laziness.
4. **Evaluator eager assumptions** — the interpreter/compiler may realize seqs in
   places (apply arg spreading, `doseq`, destructuring a seq). Audit needed.
5. **CPU-bound hangs are uninterruptible.** An infinite realization is a tight
   Janet loop with no yield points, so `ev/with-deadline` cannot truncate it
   in-process — it pins the core. This is why the suite runs each file in a
   **subprocess** (`os/spawn` + 6 s `ev/with-deadline`, then `os/proc-kill`). Phase
   5 testing must do the same (see §7).

---

## 2. Design principles (the cardinal rules)

1. **A transformer never forces its input.** It returns a LazySeq whose thunk pulls
   one element at a time via `core-first`/`core-rest`/`seq-done?`. No
   `realize-for-iteration` inside a transformer.
2. **Force only at realization boundaries.** Exactly the operations that *must* see
   all elements: `pr`/`print`/`str` rendering, `=`, `count`, `reduce`, `into`,
   `vec`/`seq`/`doall`, `doseq`, `nth`/`last` (these pull only as far as needed),
   `apply` (spreads finitely). These are allowed to loop; on a genuinely infinite
   seq they hang — matching Clojure.
3. **One-element-at-a-time, memoized.** Reuse `make-lazy-seq`/`realize-ls`; never
   re-walk. `realize-ls`'s `:jolt/pending` guard preserves self-reference.
4. **Stack safety.** A chain of N lazy wrappers must not consume N stack frames per
   element. Realize iteratively (a `while` over `realize-ls`), not by deep
   recursion through `ls-rest`. Watch `concat`/`mapcat`/`lazy-cat` especially.
5. **Multi-arity stays correct.** `map`/`mapcat` over multiple colls advance each
   input one step per output element and stop at the shortest.

---

## 3. Step-by-step implementation

Order matters: build the helper layer, then convert transformers leaf-first, then
fix boundaries, then the evaluator. Gate (§6) after **every** numbered step.

### Step 0 — Safety net
- Record the baseline: conformance 229×3, clojure-test-suite `baseline-pass=3926`,
  fixpoint stage1==2==3, self-host, all specs+unit, `lazy-seqs-spec` /
  `sequences-spec` / `transducers-spec` green.
- Build the **infinite-seq harness** first (see §6.2, "Deadlined infinite-seq
  spec") so every subsequent step is verified against hangs, not just values.
- Snapshot which clojure-test-suite files currently time out (the ~9). Save the
  list — it's the acceptance target.

### Step 1 — Lazy combinator layer
Add a small set of internal lazy builders so transformers compose uniformly,
rather than each re-implementing the thunk dance:
- `lazy-cons val thunk` → a LazySeq cell of `val` + a deferred rest.
- `lazy-from coll` → coerce any seqable to a uniform lazy view *without forcing*
  (vector/list/set/map/string/LazySeq → a LazySeq that pulls element by element).
  This is the lazy analogue of `realize-for-iteration` and the key primitive: every
  transformer takes `(lazy-from input)` and walks it with `core-first`/`core-rest`.
- `seq-done?` already exists — confirm it short-circuits without forcing the tail.
- Decide placement: the lazy machinery is host-coupled (Janet thunks) so it stays
  in `phm.janet`/`core.janet`; transformers that are already in the overlay tiers
  call these as primitives.

### Step 2 — Convert the core transformers (leaf-first)
Make each return a LazySeq over `lazy-from input`. Do them in dependency order, one
small batch per commit, each gated:
- **2a. Single-input maps/filters:** `map` (1-coll), `filter`, `remove`, `keep`,
  `map-indexed`, `keep-indexed`, `take-while`, `drop-while`, `take-nth`.
- **2b. Structural:** `cons`, `rest`/`next` over lazy, `concat`, `lazy-cat`
  (verify), `mapcat`, `cycle` (verify), `interleave`, `interpose`.
- **2c. Windowing:** `partition`, `partition-all`, `partition-by`, `dedupe`,
  `distinct`, `take`/`drop` (return LazySeq, not eager array, when input is lazy).
- **2d. Multi-input `map`/`mapcat`** over several colls (shortest-stops).
- **2e. Tree/seq:** `tree-seq`, `flatten`, `xml-seq`, `line-seq`, `sequence`,
  `iterator-seq`, `enumeration-seq`.
- For each: a transducer arity may exist (`td-*`) — leave it; only the
  collection arity changes.

### Step 3 — Realization boundaries
Audit the 57 `realize-for-iteration` call sites. Classify each as **boundary**
(keep, it must force) or **transformer leak** (remove, made lazy in Step 2):
- Boundaries that stay: `count`, `reduce`, `into`, `vec`, `seq`, `doall`, `dorun`,
  `=`/equality, `pr`/`print`/`str-render`, `sort`/`sort-by`, `reverse`, `frequencies`,
  `group-by`, `apply` arg-spread, `doseq`.
- Make sure `first`/`second`/`nth`/`last`/`take`/`get` pull **only as far as
  needed** (they must not call `realize-for-iteration`).
- `realized?` must report a LazySeq's `:realized` flag (don't force to answer).

### Step 4 — Evaluator / compiler eager assumptions
Grep the interpreter (`src/jolt/evaluator.janet`) and back end
(`src/jolt/backend.janet`, `compiler.janet`) for places that realize seqs:
- `apply` / variadic arg spreading — must finitely spread, not realize an infinite
  tail beyond the call.
- `&`-rest binding in `fn*`/`let*`/`loop*` and `destructure` — a rest param over a
  lazy seq should stay lazy, not eagerly slurp.
- `doseq`/`for` desugaring (they go through `count`/`mapcat` — verify the `for`
  comprehension stays lazy where Clojure's is).
- Any `(each x (realize ...))` in hot paths that assumes finiteness.

### Step 5 — Laziness-coupled stragglers (the deferred Phase-5 list)
From `jolt-c09` notes / MIGRATION.md: `sequence`, `sequential?`, `seqable?`,
`realized?`, `line-seq`, `rand-int`, `random-uuid`, `trampoline`, `unreduced`,
`ensure-reduced`, the transducer machinery (`cat`, `eduction`, `transduce`,
`sequence`, `halt-when`, `dedupe`/`interpose`/`keep` transducer arities). Move the
now-lazy ones to the overlay where feasible (Phase-4 style), keeping the
`Reduced`/thunk kernels native.

### Step 6 — Representation decision (DO THIS DELIBERATELY, EARLY)
Clojure: `(map inc [1 2 3])` returns a **lazy seq**, not a vector; `(seq? (map ...))`
is true, `(vector? (map ...))` is false. Jolt currently returns an eager vector
(`make-vec`) to "preserve representation". Two options:
- **(A) Full Clojure semantics:** `map`/`filter`/etc. always return a LazySeq, even
  over a vector. Most correct; **but** flips `vector?`/`seq?`/printing on a lot of
  existing results and may shift many conformance/suite assertions. Budget for the
  churn.
- **(B) Hybrid (status quo extended):** lazy over lazy/infinite input, eager
  representation-preserving over concrete finite input. Less churn, but
  `(seq? (map inc [1 2 3]))` stays wrong.
Recommend (A) for correctness, but measure the blast radius first: run conformance
+ suite with a throwaway always-lazy `map` and count newly-failing assertions
before committing to it. Whichever you pick, **write it down here and be
consistent** across all transformers.

---

## 4. Suggested commit cadence

One transformer family (a §3 sub-step) per commit. Each commit:
1. Convert the fns (overlay or core as appropriate).
2. Add infinite-seq spec cases (§6.2) + value cases.
3. Run the full gate (§6.1). Commit only if green. Push.

Mirror the Phase 4 discipline: small, gated, reversible batches.

---

## 5. Risks & gotchas

- **Uninterruptible hangs:** never probe an infinite case in-process — it pins a
  core and can't be killed by a deadline. Always go through the subprocess harness.
- **Self-reference:** `(def s (lazy-seq (cons 1 s)))` and `lazy-cat` fib rely on
  `realize-ls`'s `:jolt/pending` guard — don't bypass `realize-ls` with a
  hand-rolled force.
- **Stack overflow** from deep wrapper chains (`concat`/`mapcat`/`iterate` of
  `iterate`) — realize iteratively.
- **Double realization / side effects:** a lazy `map` fn with side effects must run
  **once per element, in order, only when forced** — assert with a counter (§7).
- **Performance:** LazySeq has per-element allocation + thunk-call overhead. Watch
  `core-bench` (`test/bench/core-bench.janet`) — the eager fast paths exist partly
  for speed. A heavy suite file slipping past the 6 s deadline = a regression
  (this already bit Phase 3's macro move).
- **Compile/self-host parity:** every behavior must hold in interpret, compile, and
  self-host (conformance runs all three). Lazy thunks are closures — verify the
  back end compiles them.
- **`chunked` seqs are out of scope** — `chunked-seq?` stays `false`. Don't emulate
  chunking; one-at-a-time is fine.

---

## 6. Testing strategy

### 6.1 Per-step gate (every commit) — same as Phase 4
```
janet test/integration/conformance-test.janet          # 229×3 (interpret/compile/self-host)
janet test/integration/bootstrap-fixpoint-test.janet   # stage1==2==3
janet test/integration/self-host-test.janet
janet test/integration/sci-bootstrap-test.janet
janet test/integration/clojure-test-suite-test.janet   # >= baseline (raise as it improves)
for f in test/spec/*.janet test/unit/*.janet; do janet "$f"; done
```

### 6.2 Deadlined infinite-seq spec (the Phase-5-specific harness)
Build this in Step 0. Plain in-process specs **cannot** test laziness — a wrong
answer hangs instead of failing. Mirror `clojure-test-suite-test.janet`'s pattern:
- A new `test/integration/lazy-infinite-test.janet` that, for each case, spawns a
  worker (`os/spawn ["janet" "test/support/lazy-eval.janet" expr]`) and waits under
  `(ev/with-deadline 5 (os/proc-wait proc))`, killing on timeout.
- A timed-out or crashed case = **FAIL** (it should have produced a value).
- Cases = the compositions that currently hang. Minimum set:
  ```
  (nth (map inc (range)) 1000)            => 1001
  (first (filter even? (drop 3 (range)))) => 4
  (take 3 (remove odd? (range)))          => (0 2 4)
  (take 3 (drop-while #(< % 5) (range)))  => (5 6 7)
  (take 4 (interleave (range) (iterate inc 10)))
  (take 3 (partition 2 (range)))          => ((0 1) (2 3) (4 5))
  (take 3 (partition-all 2 (range)))
  (take 3 (map-indexed vector (range)))
  (take 5 (distinct (cycle [1 2 1 3 1])))
  (take 3 (mapcat (fn [x] [x x]) (range)))
  (take 3 (take-nth 2 (range)))
  (take 3 (interpose :x (range)))
  (take 3 (map vector (range) (iterate inc 100)))
  (second (cons :a (range)))
  ```
  Add one row per transformer converted in Step 2.

### 6.3 Laziness assertions (side-effect counting)
For each lazy transformer, assert it realizes **only what's demanded** — values
alone don't prove laziness. Use a counter:
```clojure
(let [n (atom 0)]
  (take 3 (map (fn [x] (swap! n inc) x) (range)))
  @n)            ; => 3  (not "hang", not 1000)
```
Add these to `test/spec/lazy-seqs-spec.janet`. They run in-process safely because
they only ever force a bounded prefix.

### 6.4 Conformance extension
Add infinite-composition rows to `conformance-test.janet` (runs ×3 modes) — the
subset of §6.2 that returns a small concrete value, e.g.
`["lazy compose" "(quote (1 3 5))" "(take 3 (filter odd? (map inc (range))))"]`.
These guard interpret/compile/self-host parity.

### 6.5 Acceptance target — the timed-out suite files
The 9 files that currently time out (snapshot in Step 0:
`cycle`/`range`/transducers-over-infinite tests) should stop timing out and start
contributing passes. Each phase-5 step should monotonically reduce the timed-out
count and **raise `baseline-pass`** in `clojure-test-suite-test.janet:35`. Final
target: 0 (or near-0) timeouts and a meaningfully higher baseline.

### 6.6 Regression guards
- `core-bench` before/after (back-to-back, load-sensitive) — no large slowdown on
  the eager-collection paths.
- `lazy-seqs-spec`, `sequences-spec`, `transducers-spec` stay green every step.

---

## 7. Done criteria
- All §6.2 infinite-seq cases return correct values under the deadline (0 hangs).
- §6.3 laziness counters prove minimal realization for every converted transformer.
- Conformance 229+×3, fixpoint, self-host, sci-bootstrap all green.
- clojure-test-suite: the ~9 infinite-seq files no longer time out; `baseline-pass`
  raised to the new steady-state; no per-file 6 s timeouts introduced.
- Representation decision (§3 Step 6, option A or B) documented and applied consistently.
- `core-bench` within noise of the Phase-4 baseline.
- `bd close jolt-c09` → closes the `jolt-1j0` epic.