From 93ddf2c85ac74ce2e0b59c9508a30e08af40a2f2 Mon Sep 17 00:00:00 2001 From: Dmitri Sotnikov Date: Fri, 26 Jun 2026 04:59:52 +0000 Subject: [PATCH] Make the benchmark harness build optimized binaries on Chez (#220) bench/run.sh was Janet-era: it invoked a 'jolt' binary and set JOLT_DIRECT_LINK/JOLT_WHOLE_PROGRAM, none of which exist on Chez, where 'joltc run -m' runs fully unoptimized (direct-link and inline default off). So the suite was measuring jolt's unoptimized path. run.sh now compiles each benchmark to an optimized AOT binary (joltc build --direct-link --opt) and times it against JVM Clojure on the same portable source, auto-detecting the Chez kernel dev files like build-smoke.sh. Adds bench/deps.edn so joltc resolves the namespaces, NO_JVM to skip the reference. mandelbrot.clj dropped its jolt.png require so the JVM reference can run it; the picture demo moved to mandelbrot_png.clj (jolt-only). README scorecard refreshed with current Chez numbers and the two-regime read (compute ~8-10x substrate floor; dispatch/alloc ~120-330x architectural gaps the passes don't touch). Stale 'jolt -m' header lines point at bench/run.sh. Co-authored-by: Yogthos --- bench/.gitignore | 1 + bench/README.md | 57 ++++++++++++++++++---------- bench/binary_trees.clj | 3 +- bench/collections.clj | 2 +- bench/deps.edn | 1 + bench/dispatch.clj | 2 +- bench/fib.clj | 2 +- bench/mandelbrot.clj | 42 +++------------------ bench/mandelbrot_png.clj | 36 ++++++++++++++++++ bench/mono_dispatch.clj | 2 +- bench/run.sh | 80 +++++++++++++++++++++++++--------------- 11 files changed, 136 insertions(+), 92 deletions(-) create mode 100644 bench/.gitignore create mode 100644 bench/deps.edn create mode 100644 bench/mandelbrot_png.clj diff --git a/bench/.gitignore b/bench/.gitignore new file mode 100644 index 0000000..bd04223 --- /dev/null +++ b/bench/.gitignore @@ -0,0 +1 @@ +.cpcache/ diff --git a/bench/README.md b/bench/README.md index d4c5219..51a4669 100644 --- a/bench/README.md +++ b/bench/README.md @@ -34,34 +34,51 @@ control with record state), k-nucleotide proper. ## Holistic scorecard -`JVM=1 bench/run.sh` runs each benchmark on jolt **and** JVM Clojure and prints -the jolt/JVM ratio — the absolute-reference scorecard. As of -the broadening (2026-06-16), ratios cluster by axis: +`bench/run.sh` compiles each benchmark to an **optimized AOT binary** (`joltc build +--direct-link --opt`) and times it against JVM Clojure running the same portable +source — the jolt/JVM scorecard. jolt's optimizing passes fire only in a build; +`joltc run -m` is unoptimized, so the harness always builds. -- **pure compute** (`mandelbrot`) is the floor, ~15× — native arith - already gets jolt closest to the JVM. -- **collections** ~28×, **fib** ~37×. -- **dispatch** ~75× (megamorphic), and `mono-dispatch` is *worse* (~110×): the - JVM inline-caches a runtime-monomorphic call site to near-free, while jolt does - a full registry dispatch regardless (devirt only fires on *statically* proven - receivers, which `reduce` over a vector doesn't give). This is the signal for - the call-site inline cache. -- **allocation** (`binary-trees`) is the widest gap — but also the most inflated - by host memory pressure, so read it as "alloc is the worst axis," not a precise - multiple. Numbers are machine-specific; regenerate with `JVM=1 bench/run.sh`. +Indicative ratios (M-series, single isolated run — numbers are machine-specific, +regenerate locally). They cluster into two regimes: + +| benchmark | ratio | axis | +|---|---|---| +| `mandelbrot` | ~8× | pure float compute | +| `fib` | ~9× | call + integer arith | +| `collections` | ~9× | persistent map/vector churn | +| `dispatch` | ~130× | megamorphic protocol dispatch | +| `binary-trees` | ~140× | escaping short-lived records (allocation/GC) | +| `mono-dispatch` | ~330× | monomorphic protocol dispatch | + +- **Compute (~8–9×)** is the substrate floor: Chez is a native-compiling AOT + Scheme, not a profiling JIT, so it can't match HotSpot on hot loops. Native arith + already gets jolt closest here. +- **Dispatch & allocation (~130–330×)** are the architectural gaps. jolt does a + full protocol-registry lookup on every call; the JVM inline-caches a + runtime-monomorphic site to near-free — which is why `mono-dispatch` is *worse* + than megamorphic. devirt only fires on *statically proven* receivers (which + `reduce`/`mapv` over a heterogeneous vector never gives), so the passes don't + engage; a call-site inline cache is the missing lever. `binary-trees` nodes + escape into the tree, so scalar-replace can't remove them — this is GC pressure. +- The optimization passes move these benchmarks <10% vs the unoptimized run, so the + gaps are not a missing-flag problem; they're the dispatch/GC/JIT-floor work. ## Running ```sh -bench/run.sh # whole-program optimization on (default) -JOLT_WHOLE_PROGRAM=0 bench/run.sh # WP off, to measure what WP buys -bench/run.sh binary-trees 16 # one benchmark, custom size +bench/run.sh # full suite + JVM scorecard +bench/run.sh fib # one benchmark, default size +bench/run.sh fib 32 # one benchmark, custom size +NO_JVM=1 bench/run.sh # jolt only (skip the JVM reference) ``` +Needs Chez's kernel dev files (`libkernel.a` + `scheme.h`) and `cc` for the build, +like `jolt build`; set `JOLT_CHEZ_CSV` to override the detected csv dir. + ## A/B against a change To measure a pass, run the suite on `main`, then on the branch, back to back -(same machine, quiet) — the same protocol used for the ray tracer. Each -benchmark prints `runs: [...]` and `mean: N ms`; compare -the means. A pass is worth landing when it moves a benchmark whose axis it +(same machine, quiet). Each benchmark prints `runs: [...]` and `mean: N ms`; +compare the means. A pass is worth landing when it moves a benchmark whose axis it targets, even if the ray tracer stays flat. diff --git a/bench/binary_trees.clj b/bench/binary_trees.clj index e3f580e..28428ec 100644 --- a/bench/binary_trees.clj +++ b/bench/binary_trees.clj @@ -4,8 +4,7 @@ ;; targets and the ray tracer never exercises (~7% alloc). ;; ;; Portable Clojure: runs on jolt and JVM Clojure for cross-impl comparison. -;; jolt -m binary-trees 14 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1) -;; clojure -M -m binary-trees 14 +;; bench/run.sh binary-trees 14 (ns binary-trees) (defrecord Node [left right]) diff --git a/bench/collections.clj b/bench/collections.clj index 16e9807..2c0304d 100644 --- a/bench/collections.clj +++ b/bench/collections.clj @@ -5,7 +5,7 @@ ;; records, no collections in the hot loop) doesn't touch. ;; ;; Portable Clojure (jolt + JVM Clojure). -;; jolt -m collections 200000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1) +;; bench/run.sh collections 200000 (ns collections) ;; map churn: accumulate a frequency map over a stream of keys, then sum it back diff --git a/bench/deps.edn b/bench/deps.edn new file mode 100644 index 0000000..5837a2a --- /dev/null +++ b/bench/deps.edn @@ -0,0 +1 @@ +{:paths ["."]} diff --git a/bench/dispatch.clj b/bench/dispatch.clj index f6decc0..d60527b 100644 --- a/bench/dispatch.clj +++ b/bench/dispatch.clj @@ -6,7 +6,7 @@ ;; float-math cost (devirt measured FLAT there). ;; ;; Portable Clojure (jolt + JVM Clojure). -;; jolt -m dispatch 20000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1) +;; bench/run.sh dispatch 20000 (ns dispatch) (defprotocol Shape diff --git a/bench/fib.clj b/bench/fib.clj index 6715e58..2457a6e 100644 --- a/bench/fib.clj +++ b/bench/fib.clj @@ -4,7 +4,7 @@ ;; single-call-site / small-fn inlining and self-call direct-linking. ;; ;; Portable Clojure (jolt + JVM Clojure). -;; jolt -m fib 32 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1) +;; bench/run.sh fib 32 (ns fib) (defn fib [n] diff --git a/bench/mandelbrot.clj b/bench/mandelbrot.clj index 0a715d9..0367f02 100644 --- a/bench/mandelbrot.clj +++ b/bench/mandelbrot.clj @@ -5,10 +5,10 @@ ;; on (where devirt/alloc passes measured flat), so it tracks native-arith codegen ;; and loop quality directly. ;; -;; Portable Clojure (jolt + JVM Clojure). -;; jolt -m mandelbrot 1000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1) -(ns mandelbrot - (:require [jolt.png :as png])) +;; Portable Clojure (jolt + JVM Clojure). The jolt.png picture demo lives in +;; mandelbrot_png.clj so this file stays portable for the JVM reference run. +;; bench/run.sh mandelbrot 1000 +(ns mandelbrot) (defn count-point [cr ci cap] (loop [i 0 zr 0.0 zi 0.0] @@ -32,35 +32,6 @@ (recur (inc y) (+ acc row))) acc)))) -;; --- PNG demo (jolt.png) -------------------------------------------------- -;; Render a real picture of the set, reusing count-point as the kernel. `render` -;; is a separate -main subcommand so the numeric-arg bench path is untouched. - -(defn- color - "Escape-iteration count -> RGB. In-set points (n>=cap) are black; faster - escapes run through a warm gradient." - [n cap] - (if (>= n cap) - [0 0 0] - (let [t (/ (double n) cap)] - [(int (* 255 (min 1.0 (* 3.0 t)))) - (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.33)))))) - (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.66))))))]))) - -(defn render! - "Render a size×size view of the Mandelbrot set to a PNG at path." - [path size] - (let [w size h size cap 1000 - img (png/image w h)] - (doseq [py (range h)] - (doseq [px (range w)] - (let [cr (- (* 3.5 (/ (double px) w)) 2.5) ; real ∈ [-2.5, 1.0] - ci (- (* 2.8 (/ (double py) h)) 1.4) ; imag ∈ [-1.4, 1.4] - [r g b] (color (count-point cr ci cap) cap)] - (png/put! img r g b)))) - (png/write img w h path) - (println "wrote" path (str w "×" h ", cap " cap)))) - (defn- run-bench [args] (let [n (if (seq args) (Integer/parseInt (first args)) 1000)] (dotimes [_ 2] (run (quot n 2))) ; warmup @@ -78,7 +49,4 @@ (println "mean:" (/ (Math/round (* mean 10.0)) 10.0) "ms")))) (defn -main [& args] - (if (= (first args) "render") - (render! (or (second args) "mandelbrot.png") - (if (nth args 2 nil) (Integer/parseInt (nth args 2)) 600)) - (run-bench args))) + (run-bench args)) diff --git a/bench/mandelbrot_png.clj b/bench/mandelbrot_png.clj new file mode 100644 index 0000000..f73d441 --- /dev/null +++ b/bench/mandelbrot_png.clj @@ -0,0 +1,36 @@ +;; mandelbrot picture demo — renders a real image of the set to a PNG via +;; jolt.png (FFI), reusing mandelbrot/count-point as the kernel. jolt-only (the +;; benchmark in mandelbrot.clj stays portable for the JVM reference). +;; joltc run -m mandelbrot-png [path] [size] +(ns mandelbrot-png + (:require [mandelbrot :as m] + [jolt.png :as png])) + +(defn- color + "Escape-iteration count -> RGB. In-set points (n>=cap) are black; faster + escapes run through a warm gradient." + [n cap] + (if (>= n cap) + [0 0 0] + (let [t (/ (double n) cap)] + [(int (* 255 (min 1.0 (* 3.0 t)))) + (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.33)))))) + (int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.66))))))]))) + +(defn render! + "Render a size×size view of the Mandelbrot set to a PNG at path." + [path size] + (let [w size h size cap 1000 + img (png/image w h)] + (doseq [py (range h)] + (doseq [px (range w)] + (let [cr (- (* 3.5 (/ (double px) w)) 2.5) ; real ∈ [-2.5, 1.0] + ci (- (* 2.8 (/ (double py) h)) 1.4) ; imag ∈ [-1.4, 1.4] + [r g b] (color (m/count-point cr ci cap) cap)] + (png/put! img r g b)))) + (png/write img w h path) + (println "wrote" path (str w "×" h ", cap " cap)))) + +(defn -main [& args] + (render! (or (first args) "mandelbrot.png") + (if (second args) (Integer/parseInt (second args)) 600))) diff --git a/bench/mono_dispatch.clj b/bench/mono_dispatch.clj index 0755acb..dba72dd 100644 --- a/bench/mono_dispatch.clj +++ b/bench/mono_dispatch.clj @@ -5,7 +5,7 @@ ;; monomorphic dispatch gets to a direct call. Same per-call work as `dispatch`. ;; ;; Portable Clojure (jolt + JVM Clojure). -;; jolt -m mono-dispatch 20000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1) +;; bench/run.sh mono-dispatch 20000 (ns mono-dispatch) (defprotocol Shape diff --git a/bench/run.sh b/bench/run.sh index 63d77af..73db4cc 100755 --- a/bench/run.sh +++ b/bench/run.sh @@ -1,48 +1,70 @@ #!/bin/sh -# Run the jolt benchmark suite and print mean ms per benchmark. +# Run the jolt benchmark suite against JVM Clojure and print a jolt/JVM scorecard. # -# Each benchmark isolates an axis the ray tracer (float-compute-bound) doesn't -# capture — see README.md. Run back-to-back against `main` to measure a pass's -# impact. +# jolt's optimizing passes (direct-linking, inlining, scalar-replace, whole-program +# inference) fire only in an AOT BUILD — `joltc run -m` is unoptimized — so each +# benchmark is compiled to an optimized standalone binary and timed. JVM Clojure +# runs the same portable source for the absolute reference. Each benchmark prints +# `runs: [...]` and `mean: N ms`; the table shows the means and the jolt/JVM ratio. # -# bench/run.sh # default sizes, whole-program optimization on -# JOLT_WHOLE_PROGRAM=0 bench/run.sh # compare with WP off -# bench/run.sh binary-trees # one benchmark +# bench/run.sh # full suite + JVM scorecard +# bench/run.sh fib # one benchmark, default size +# bench/run.sh fib 32 # one benchmark, custom size +# NO_JVM=1 bench/run.sh # jolt only (skip the JVM reference) # -# Needs `jolt` on PATH (build with `jpm build`; export PATH="$PWD/build:$PATH"). +# Building needs Chez's kernel dev files (libkernel.a + scheme.h) and a C compiler, +# the same as `jolt build`; set JOLT_CHEZ_CSV to override the detected csv dir. set -e cd "$(dirname "$0")" +root="$(cd .. && pwd)" +joltc="$root/bin/joltc" +export JOLT_PWD="$PWD" -export JOLT_DIRECT_LINK="${JOLT_DIRECT_LINK:-1}" -export JOLT_WHOLE_PROGRAM="${JOLT_WHOLE_PROGRAM:-1}" -export JOLT_APP_PATHS="$PWD" -export JOLT_PATH="$PWD" +# Locate Chez's kernel dev files for the optimized build (as build-smoke.sh does). +csv="$JOLT_CHEZ_CSV" +if [ -z "$csv" ]; then + chez_bin="$(command -v chez || command -v scheme || command -v petite || true)" + if [ -n "$chez_bin" ]; then + base="$(cd "$(dirname "$chez_bin")/.." 2>/dev/null && pwd)" + for d in "$base"/lib/csv*/*/; do + [ -f "${d}libkernel.a" ] && csv="${d%/}" && break + done + fi +fi +if [ -z "$csv" ] || [ ! -f "$csv/libkernel.a" ] || [ ! -f "$csv/scheme.h" ] || ! command -v cc >/dev/null 2>&1; then + echo "error: the optimized build needs Chez kernel dev files (libkernel.a + scheme.h) and cc." >&2 + echo " set JOLT_CHEZ_CSV to the csv dir, e.g. \$(brew --prefix chezscheme)/lib/csv*/." >&2 + exit 1 +fi +export JOLT_CHEZ_CSV="$csv" -# name:default-arg (arg sized to run in a few seconds each). Axes: allocation -# (binary-trees), megamorphic vs monomorphic dispatch, persistent-collection -# churn (collections — now O(log n) via the HAMT, so sized up), pure -# float compute (mandelbrot), call+arith recursion (fib). -BENCHES="binary-trees:14 dispatch:2000 mono-dispatch:2000 collections:30000 mandelbrot:200 fib:30" +bindir="$(mktemp -d)" +trap 'rm -rf "$bindir"' EXIT + +# name:default-arg, each sized to run in a few seconds. Axes: see README.md. +BENCHES="fib:30 mandelbrot:200 collections:30000 mono-dispatch:2000 dispatch:2000 binary-trees:14" -# JVM=1 also runs each bench on JVM Clojure and prints a jolt/JVM ratio — the -# holistic absolute-reference scorecard for the optimization work. run_one() { ns="${1%%:*}"; arg="${2:-${1##*:}}" - jmean=$(jolt -m "$ns" "$arg" 2>&1 | awk '/^mean:/{print $2}') - if [ -n "$JVM" ]; then - vmean=$(clojure -Sdeps '{:paths ["."]}' -M -m "$ns" "$arg" 2>&1 | awk '/^mean:/{print $2}') - ratio=$(awk "BEGIN{ if ($vmean+0>0) printf \"%.1f\", ($jmean+0)/($vmean+0); else printf \"-\" }") - printf '%-16s jolt %9s ms jvm %8s ms %sx\n' "$ns" "${jmean:--}" "${vmean:--}" "$ratio" + if ! "$joltc" build -m "$ns" -o "$bindir/$ns" --direct-link --opt >/dev/null 2>&1; then + printf '%-16s jolt build FAILED\n' "$ns"; return + fi + jmean=$("$bindir/$ns" "$arg" 2>/dev/null | awk '/^mean:/{print $2}') + if [ -z "$NO_JVM" ]; then + vmean=$(clojure -Sdeps '{:paths ["."]}' -M -m "$ns" "$arg" 2>/dev/null | awk '/^mean:/{print $2}') + ratio=$(awk "BEGIN{ if (\"$vmean\"+0>0 && \"$jmean\"+0>0) printf \"%.1fx\", (\"$jmean\"+0)/(\"$vmean\"+0); else printf \"-\" }") + printf '%-16s jolt %9s ms jvm %8s ms %s\n' "$ns" "${jmean:--}" "${vmean:--}" "$ratio" else - printf '%-16s %9s ms\n' "$ns" "${jmean:--}" + printf '%-16s jolt %9s ms\n' "$ns" "${jmean:--}" fi } if [ -n "$1" ]; then - for spec in $BENCHES; do - [ "${spec%%:*}" = "$1" ] && run_one "$spec" "$2" - done + spec="" + for s in $BENCHES; do [ "${s%%:*}" = "$1" ] && spec="$s"; done + [ -n "$spec" ] || { echo "unknown benchmark: $1 (have: ${BENCHES})" >&2; exit 1; } + run_one "$spec" "$2" else - echo "jolt benchmark suite (WP=$JOLT_WHOLE_PROGRAM${JVM:+, vs JVM Clojure})" + echo "jolt benchmark suite — optimized AOT binaries${NO_JVM:+ }${NO_JVM:-, vs JVM Clojure}" for spec in $BENCHES; do run_one "$spec"; done fi