Make the benchmark harness build optimized binaries on Chez (#220)

bench/run.sh was Janet-era: it invoked a 'jolt' binary and set
JOLT_DIRECT_LINK/JOLT_WHOLE_PROGRAM, none of which exist on Chez, where
'joltc run -m' runs fully unoptimized (direct-link and inline default off). So
the suite was measuring jolt's unoptimized path.

run.sh now compiles each benchmark to an optimized AOT binary (joltc build
--direct-link --opt) and times it against JVM Clojure on the same portable
source, auto-detecting the Chez kernel dev files like build-smoke.sh. Adds
bench/deps.edn so joltc resolves the namespaces, NO_JVM to skip the reference.

mandelbrot.clj dropped its jolt.png require so the JVM reference can run it; the
picture demo moved to mandelbrot_png.clj (jolt-only). README scorecard refreshed
with current Chez numbers and the two-regime read (compute ~8-10x substrate floor;
dispatch/alloc ~120-330x architectural gaps the passes don't touch). Stale
'jolt -m' header lines point at bench/run.sh.

Co-authored-by: Yogthos <yogthos@gmail.com>

bench/.gitignore

@@ -0,0 +1 @@
+.cpcache/

bench/README.md

@@ -34,34 +34,51 @@ control with record state), k-nucleotide proper.
 
 ## Holistic scorecard
 
-`JVM=1 bench/run.sh` runs each benchmark on jolt **and** JVM Clojure and prints
+`bench/run.sh` compiles each benchmark to an **optimized AOT binary** (`joltc build
-the jolt/JVM ratio — the absolute-reference scorecard. As of
+--direct-link --opt`) and times it against JVM Clojure running the same portable
-the broadening (2026-06-16), ratios cluster by axis:
+source — the jolt/JVM scorecard. jolt's optimizing passes fire only in a build;
+`joltc run -m` is unoptimized, so the harness always builds.
 
-- **pure compute** (`mandelbrot`) is the floor, ~15× — native arith
+Indicative ratios (M-series, single isolated run — numbers are machine-specific,
-  already gets jolt closest to the JVM.
+regenerate locally). They cluster into two regimes:
-- **collections** ~28×, **fib** ~37×.
+
-- **dispatch** ~75× (megamorphic), and `mono-dispatch` is *worse* (~110×): the
+| benchmark | ratio | axis |
-  JVM inline-caches a runtime-monomorphic call site to near-free, while jolt does
+|---|---|---|
-  a full registry dispatch regardless (devirt only fires on *statically* proven
+| `mandelbrot` | ~8× | pure float compute |
-  receivers, which `reduce` over a vector doesn't give). This is the signal for
+| `fib` | ~9× | call + integer arith |
-  the call-site inline cache.
+| `collections` | ~9× | persistent map/vector churn |
-- **allocation** (`binary-trees`) is the widest gap — but also the most inflated
+| `dispatch` | ~130× | megamorphic protocol dispatch |
-  by host memory pressure, so read it as "alloc is the worst axis," not a precise
+| `binary-trees` | ~140× | escaping short-lived records (allocation/GC) |
-  multiple. Numbers are machine-specific; regenerate with `JVM=1 bench/run.sh`.
+| `mono-dispatch` | ~330× | monomorphic protocol dispatch |
+
+- **Compute (~8–9×)** is the substrate floor: Chez is a native-compiling AOT
+  Scheme, not a profiling JIT, so it can't match HotSpot on hot loops. Native arith
+  already gets jolt closest here.
+- **Dispatch & allocation (~130–330×)** are the architectural gaps. jolt does a
+  full protocol-registry lookup on every call; the JVM inline-caches a
+  runtime-monomorphic site to near-free — which is why `mono-dispatch` is *worse*
+  than megamorphic. devirt only fires on *statically proven* receivers (which
+  `reduce`/`mapv` over a heterogeneous vector never gives), so the passes don't
+  engage; a call-site inline cache is the missing lever. `binary-trees` nodes
+  escape into the tree, so scalar-replace can't remove them — this is GC pressure.
+- The optimization passes move these benchmarks <10% vs the unoptimized run, so the
+  gaps are not a missing-flag problem; they're the dispatch/GC/JIT-floor work.
 
 ## Running
 
 ```sh
-bench/run.sh                      # whole-program optimization on (default)
+bench/run.sh                 # full suite + JVM scorecard
-JOLT_WHOLE_PROGRAM=0 bench/run.sh # WP off, to measure what WP buys
+bench/run.sh fib             # one benchmark, default size
-bench/run.sh binary-trees 16      # one benchmark, custom size
+bench/run.sh fib 32          # one benchmark, custom size
+NO_JVM=1 bench/run.sh        # jolt only (skip the JVM reference)
 ```
 
+Needs Chez's kernel dev files (`libkernel.a` + `scheme.h`) and `cc` for the build,
+like `jolt build`; set `JOLT_CHEZ_CSV` to override the detected csv dir.
+
 ## A/B against a change
 
 To measure a pass, run the suite on `main`, then on the branch, back to back
-(same machine, quiet) — the same protocol used for the ray tracer. Each
+(same machine, quiet). Each benchmark prints `runs: [...]` and `mean: N ms`;
-benchmark prints `runs: [...]` and `mean: N ms`; compare
+compare the means. A pass is worth landing when it moves a benchmark whose axis it
-the means. A pass is worth landing when it moves a benchmark whose axis it
 targets, even if the ray tracer stays flat.

bench/binary_trees.clj

@@ -4,8 +4,7 @@
 ;; targets and the ray tracer never exercises (~7% alloc).
 ;;
 ;; Portable Clojure: runs on jolt and JVM Clojure for cross-impl comparison.
-;;   jolt -m binary-trees 14        (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
+;;   bench/run.sh binary-trees 14
-;;   clojure -M -m binary-trees 14
 (ns binary-trees)
 
 (defrecord Node [left right])

bench/collections.clj

@@ -5,7 +5,7 @@
 ;; records, no collections in the hot loop) doesn't touch.
 ;;
 ;; Portable Clojure (jolt + JVM Clojure).
-;;   jolt -m collections 200000     (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
+;;   bench/run.sh collections 200000
 (ns collections)
 
 ;; map churn: accumulate a frequency map over a stream of keys, then sum it back

bench/deps.edn

@@ -0,0 +1 @@
+{:paths ["."]}

bench/dispatch.clj

@@ -6,7 +6,7 @@
 ;; float-math cost (devirt measured FLAT there).
 ;;
 ;; Portable Clojure (jolt + JVM Clojure).
-;;   jolt -m dispatch 20000     (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
+;;   bench/run.sh dispatch 20000
 (ns dispatch)
 
 (defprotocol Shape

bench/fib.clj

@@ -4,7 +4,7 @@
 ;; single-call-site / small-fn inlining and self-call direct-linking.
 ;;
 ;; Portable Clojure (jolt + JVM Clojure).
-;;   jolt -m fib 32     (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
+;;   bench/run.sh fib 32
 (ns fib)
 
 (defn fib [n]

bench/mandelbrot.clj

@@ -5,10 +5,10 @@
 ;; on (where devirt/alloc passes measured flat), so it tracks native-arith codegen
 ;; and loop quality directly.
 ;;
-;; Portable Clojure (jolt + JVM Clojure).
+;; Portable Clojure (jolt + JVM Clojure). The jolt.png picture demo lives in
-;;   jolt -m mandelbrot 1000     (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
+;; mandelbrot_png.clj so this file stays portable for the JVM reference run.
-(ns mandelbrot
+;;   bench/run.sh mandelbrot 1000
-  (:require [jolt.png :as png]))
+(ns mandelbrot)
 
 (defn count-point [cr ci cap]
   (loop [i 0 zr 0.0 zi 0.0]
@@ -32,35 +32,6 @@
           (recur (inc y) (+ acc row)))
         acc))))
 
-;; --- PNG demo (jolt.png) --------------------------------------------------
-;; Render a real picture of the set, reusing count-point as the kernel. `render`
-;; is a separate -main subcommand so the numeric-arg bench path is untouched.
-
-(defn- color
-  "Escape-iteration count -> RGB. In-set points (n>=cap) are black; faster
-  escapes run through a warm gradient."
-  [n cap]
-  (if (>= n cap)
-    [0 0 0]
-    (let [t (/ (double n) cap)]
-      [(int (* 255 (min 1.0 (* 3.0 t))))
-       (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.33))))))
-       (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.66))))))])))
-
-(defn render!
-  "Render a size×size view of the Mandelbrot set to a PNG at path."
-  [path size]
-  (let [w size h size cap 1000
-        img (png/image w h)]
-    (doseq [py (range h)]
-      (doseq [px (range w)]
-        (let [cr (- (* 3.5 (/ (double px) w)) 2.5)        ; real ∈ [-2.5, 1.0]
-              ci (- (* 2.8 (/ (double py) h)) 1.4)        ; imag ∈ [-1.4, 1.4]
-              [r g b] (color (count-point cr ci cap) cap)]
-          (png/put! img r g b))))
-    (png/write img w h path)
-    (println "wrote" path (str w "×" h ", cap " cap))))
-
 (defn- run-bench [args]
   (let [n (if (seq args) (Integer/parseInt (first args)) 1000)]
     (dotimes [_ 2] (run (quot n 2)))                     ; warmup
@@ -78,7 +49,4 @@
       (println "mean:" (/ (Math/round (* mean 10.0)) 10.0) "ms"))))
 
 (defn -main [& args]
-  (if (= (first args) "render")
+  (run-bench args))
-    (render! (or (second args) "mandelbrot.png")
-             (if (nth args 2 nil) (Integer/parseInt (nth args 2)) 600))
-    (run-bench args)))

bench/mandelbrot_png.clj

@@ -0,0 +1,36 @@
+;; mandelbrot picture demo — renders a real image of the set to a PNG via
+;; jolt.png (FFI), reusing mandelbrot/count-point as the kernel. jolt-only (the
+;; benchmark in mandelbrot.clj stays portable for the JVM reference).
+;;   joltc run -m mandelbrot-png [path] [size]
+(ns mandelbrot-png
+  (:require [mandelbrot :as m]
+            [jolt.png :as png]))
+
+(defn- color
+  "Escape-iteration count -> RGB. In-set points (n>=cap) are black; faster
+  escapes run through a warm gradient."
+  [n cap]
+  (if (>= n cap)
+    [0 0 0]
+    (let [t (/ (double n) cap)]
+      [(int (* 255 (min 1.0 (* 3.0 t))))
+       (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.33))))))
+       (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.66))))))])))
+
+(defn render!
+  "Render a size×size view of the Mandelbrot set to a PNG at path."
+  [path size]
+  (let [w size h size cap 1000
+        img (png/image w h)]
+    (doseq [py (range h)]
+      (doseq [px (range w)]
+        (let [cr (- (* 3.5 (/ (double px) w)) 2.5)        ; real ∈ [-2.5, 1.0]
+              ci (- (* 2.8 (/ (double py) h)) 1.4)        ; imag ∈ [-1.4, 1.4]
+              [r g b] (color (m/count-point cr ci cap) cap)]
+          (png/put! img r g b))))
+    (png/write img w h path)
+    (println "wrote" path (str w "×" h ", cap " cap))))
+
+(defn -main [& args]
+  (render! (or (first args) "mandelbrot.png")
+           (if (second args) (Integer/parseInt (second args)) 600)))

bench/mono_dispatch.clj

@@ -5,7 +5,7 @@
 ;; monomorphic dispatch gets to a direct call. Same per-call work as `dispatch`.
 ;;
 ;; Portable Clojure (jolt + JVM Clojure).
-;;   jolt -m mono-dispatch 20000     (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
+;;   bench/run.sh mono-dispatch 20000
 (ns mono-dispatch)
 
 (defprotocol Shape

bench/run.sh

@@ -1,48 +1,70 @@
 #!/bin/sh
-# Run the jolt benchmark suite and print mean ms per benchmark.
+# Run the jolt benchmark suite against JVM Clojure and print a jolt/JVM scorecard.
 #
-# Each benchmark isolates an axis the ray tracer (float-compute-bound) doesn't
+# jolt's optimizing passes (direct-linking, inlining, scalar-replace, whole-program
-# capture — see README.md. Run back-to-back against `main` to measure a pass's
+# inference) fire only in an AOT BUILD — `joltc run -m` is unoptimized — so each
-# impact.
+# benchmark is compiled to an optimized standalone binary and timed. JVM Clojure
+# runs the same portable source for the absolute reference. Each benchmark prints
+# `runs: [...]` and `mean: N ms`; the table shows the means and the jolt/JVM ratio.
 #
-#   bench/run.sh                 # default sizes, whole-program optimization on
+#   bench/run.sh                 # full suite + JVM scorecard
-#   JOLT_WHOLE_PROGRAM=0 bench/run.sh   # compare with WP off
+#   bench/run.sh fib             # one benchmark, default size
-#   bench/run.sh binary-trees    # one benchmark
+#   bench/run.sh fib 32          # one benchmark, custom size
+#   NO_JVM=1 bench/run.sh        # jolt only (skip the JVM reference)
 #
-# Needs `jolt` on PATH (build with `jpm build`; export PATH="$PWD/build:$PATH").
+# Building needs Chez's kernel dev files (libkernel.a + scheme.h) and a C compiler,
+# the same as `jolt build`; set JOLT_CHEZ_CSV to override the detected csv dir.
 set -e
 cd "$(dirname "$0")"
+root="$(cd .. && pwd)"
+joltc="$root/bin/joltc"
+export JOLT_PWD="$PWD"
 
-export JOLT_DIRECT_LINK="${JOLT_DIRECT_LINK:-1}"
+# Locate Chez's kernel dev files for the optimized build (as build-smoke.sh does).
-export JOLT_WHOLE_PROGRAM="${JOLT_WHOLE_PROGRAM:-1}"
+csv="$JOLT_CHEZ_CSV"
-export JOLT_APP_PATHS="$PWD"
+if [ -z "$csv" ]; then
-export JOLT_PATH="$PWD"
+  chez_bin="$(command -v chez || command -v scheme || command -v petite || true)"
+  if [ -n "$chez_bin" ]; then
+    base="$(cd "$(dirname "$chez_bin")/.." 2>/dev/null && pwd)"
+    for d in "$base"/lib/csv*/*/; do
+      [ -f "${d}libkernel.a" ] && csv="${d%/}" && break
+    done
+  fi
+fi
+if [ -z "$csv" ] || [ ! -f "$csv/libkernel.a" ] || [ ! -f "$csv/scheme.h" ] || ! command -v cc >/dev/null 2>&1; then
+  echo "error: the optimized build needs Chez kernel dev files (libkernel.a + scheme.h) and cc." >&2
+  echo "       set JOLT_CHEZ_CSV to the csv dir, e.g. \$(brew --prefix chezscheme)/lib/csv*/<machine>." >&2
+  exit 1
+fi
+export JOLT_CHEZ_CSV="$csv"
 
-# name:default-arg  (arg sized to run in a few seconds each). Axes: allocation
+bindir="$(mktemp -d)"
-# (binary-trees), megamorphic vs monomorphic dispatch, persistent-collection
+trap 'rm -rf "$bindir"' EXIT
-# churn (collections — now O(log n) via the HAMT, so sized up), pure
+
-# float compute (mandelbrot), call+arith recursion (fib).
+# name:default-arg, each sized to run in a few seconds. Axes: see README.md.
-BENCHES="binary-trees:14 dispatch:2000 mono-dispatch:2000 collections:30000 mandelbrot:200 fib:30"
+BENCHES="fib:30 mandelbrot:200 collections:30000 mono-dispatch:2000 dispatch:2000 binary-trees:14"
 
-# JVM=1 also runs each bench on JVM Clojure and prints a jolt/JVM ratio — the
-# holistic absolute-reference scorecard for the optimization work.
 run_one() {
   ns="${1%%:*}"; arg="${2:-${1##*:}}"
-  jmean=$(jolt -m "$ns" "$arg" 2>&1 | awk '/^mean:/{print $2}')
+  if ! "$joltc" build -m "$ns" -o "$bindir/$ns" --direct-link --opt >/dev/null 2>&1; then
-  if [ -n "$JVM" ]; then
+    printf '%-16s  jolt build FAILED\n' "$ns"; return
-    vmean=$(clojure -Sdeps '{:paths ["."]}' -M -m "$ns" "$arg" 2>&1 | awk '/^mean:/{print $2}')
+  fi
-    ratio=$(awk "BEGIN{ if ($vmean+0>0) printf \"%.1f\", ($jmean+0)/($vmean+0); else printf \"-\" }")
+  jmean=$("$bindir/$ns" "$arg" 2>/dev/null | awk '/^mean:/{print $2}')
-    printf '%-16s jolt %9s ms   jvm %8s ms   %sx\n' "$ns" "${jmean:--}" "${vmean:--}" "$ratio"
+  if [ -z "$NO_JVM" ]; then
+    vmean=$(clojure -Sdeps '{:paths ["."]}' -M -m "$ns" "$arg" 2>/dev/null | awk '/^mean:/{print $2}')
+    ratio=$(awk "BEGIN{ if (\"$vmean\"+0>0 && \"$jmean\"+0>0) printf \"%.1fx\", (\"$jmean\"+0)/(\"$vmean\"+0); else printf \"-\" }")
+    printf '%-16s jolt %9s ms   jvm %8s ms   %s\n' "$ns" "${jmean:--}" "${vmean:--}" "$ratio"
   else
-    printf '%-16s %9s ms\n' "$ns" "${jmean:--}"
+    printf '%-16s jolt %9s ms\n' "$ns" "${jmean:--}"
   fi
 }
 
 if [ -n "$1" ]; then
-  for spec in $BENCHES; do
+  spec=""
-    [ "${spec%%:*}" = "$1" ] && run_one "$spec" "$2"
+  for s in $BENCHES; do [ "${s%%:*}" = "$1" ] && spec="$s"; done
-  done
+  [ -n "$spec" ] || { echo "unknown benchmark: $1 (have: ${BENCHES})" >&2; exit 1; }
+  run_one "$spec" "$2"
 else
-  echo "jolt benchmark suite (WP=$JOLT_WHOLE_PROGRAM${JVM:+, vs JVM Clojure})"
+  echo "jolt benchmark suite — optimized AOT binaries${NO_JVM:+ }${NO_JVM:-, vs JVM Clojure}"
   for spec in $BENCHES; do run_one "$spec"; done
 fi