Make the benchmark harness build optimized binaries on Chez (#220)
bench/run.sh was Janet-era: it invoked a 'jolt' binary and set JOLT_DIRECT_LINK/JOLT_WHOLE_PROGRAM, none of which exist on Chez, where 'joltc run -m' runs fully unoptimized (direct-link and inline default off). So the suite was measuring jolt's unoptimized path. run.sh now compiles each benchmark to an optimized AOT binary (joltc build --direct-link --opt) and times it against JVM Clojure on the same portable source, auto-detecting the Chez kernel dev files like build-smoke.sh. Adds bench/deps.edn so joltc resolves the namespaces, NO_JVM to skip the reference. mandelbrot.clj dropped its jolt.png require so the JVM reference can run it; the picture demo moved to mandelbrot_png.clj (jolt-only). README scorecard refreshed with current Chez numbers and the two-regime read (compute ~8-10x substrate floor; dispatch/alloc ~120-330x architectural gaps the passes don't touch). Stale 'jolt -m' header lines point at bench/run.sh. Co-authored-by: Yogthos <yogthos@gmail.com>
This commit is contained in:
Dmitri Sotnikov
GitHub
parent
f3084f8043
commit
93ddf2c85a
11 changed files with 136 additions and 92 deletions
1
bench/.gitignore
vendored
Normal file
1
bench/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
|||
.cpcache/
|
||||
|
|
@ -34,34 +34,51 @@ control with record state), k-nucleotide proper.
|
|||
|
||||
## Holistic scorecard
|
||||
|
||||
`JVM=1 bench/run.sh` runs each benchmark on jolt **and** JVM Clojure and prints
|
||||
the jolt/JVM ratio — the absolute-reference scorecard. As of
|
||||
the broadening (2026-06-16), ratios cluster by axis:
|
||||
`bench/run.sh` compiles each benchmark to an **optimized AOT binary** (`joltc build
|
||||
--direct-link --opt`) and times it against JVM Clojure running the same portable
|
||||
source — the jolt/JVM scorecard. jolt's optimizing passes fire only in a build;
|
||||
`joltc run -m` is unoptimized, so the harness always builds.
|
||||
|
||||
- **pure compute** (`mandelbrot`) is the floor, ~15× — native arith
|
||||
already gets jolt closest to the JVM.
|
||||
- **collections** ~28×, **fib** ~37×.
|
||||
- **dispatch** ~75× (megamorphic), and `mono-dispatch` is *worse* (~110×): the
|
||||
JVM inline-caches a runtime-monomorphic call site to near-free, while jolt does
|
||||
a full registry dispatch regardless (devirt only fires on *statically* proven
|
||||
receivers, which `reduce` over a vector doesn't give). This is the signal for
|
||||
the call-site inline cache.
|
||||
- **allocation** (`binary-trees`) is the widest gap — but also the most inflated
|
||||
by host memory pressure, so read it as "alloc is the worst axis," not a precise
|
||||
multiple. Numbers are machine-specific; regenerate with `JVM=1 bench/run.sh`.
|
||||
Indicative ratios (M-series, single isolated run — numbers are machine-specific,
|
||||
regenerate locally). They cluster into two regimes:
|
||||
|
||||
| benchmark | ratio | axis |
|
||||
|---|---|---|
|
||||
| `mandelbrot` | ~8× | pure float compute |
|
||||
| `fib` | ~9× | call + integer arith |
|
||||
| `collections` | ~9× | persistent map/vector churn |
|
||||
| `dispatch` | ~130× | megamorphic protocol dispatch |
|
||||
| `binary-trees` | ~140× | escaping short-lived records (allocation/GC) |
|
||||
| `mono-dispatch` | ~330× | monomorphic protocol dispatch |
|
||||
|
||||
- **Compute (~8–9×)** is the substrate floor: Chez is a native-compiling AOT
|
||||
Scheme, not a profiling JIT, so it can't match HotSpot on hot loops. Native arith
|
||||
already gets jolt closest here.
|
||||
- **Dispatch & allocation (~130–330×)** are the architectural gaps. jolt does a
|
||||
full protocol-registry lookup on every call; the JVM inline-caches a
|
||||
runtime-monomorphic site to near-free — which is why `mono-dispatch` is *worse*
|
||||
than megamorphic. devirt only fires on *statically proven* receivers (which
|
||||
`reduce`/`mapv` over a heterogeneous vector never gives), so the passes don't
|
||||
engage; a call-site inline cache is the missing lever. `binary-trees` nodes
|
||||
escape into the tree, so scalar-replace can't remove them — this is GC pressure.
|
||||
- The optimization passes move these benchmarks <10% vs the unoptimized run, so the
|
||||
gaps are not a missing-flag problem; they're the dispatch/GC/JIT-floor work.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
bench/run.sh # whole-program optimization on (default)
|
||||
JOLT_WHOLE_PROGRAM=0 bench/run.sh # WP off, to measure what WP buys
|
||||
bench/run.sh binary-trees 16 # one benchmark, custom size
|
||||
bench/run.sh # full suite + JVM scorecard
|
||||
bench/run.sh fib # one benchmark, default size
|
||||
bench/run.sh fib 32 # one benchmark, custom size
|
||||
NO_JVM=1 bench/run.sh # jolt only (skip the JVM reference)
|
||||
```
|
||||
|
||||
Needs Chez's kernel dev files (`libkernel.a` + `scheme.h`) and `cc` for the build,
|
||||
like `jolt build`; set `JOLT_CHEZ_CSV` to override the detected csv dir.
|
||||
|
||||
## A/B against a change
|
||||
|
||||
To measure a pass, run the suite on `main`, then on the branch, back to back
|
||||
(same machine, quiet) — the same protocol used for the ray tracer. Each
|
||||
benchmark prints `runs: [...]` and `mean: N ms`; compare
|
||||
the means. A pass is worth landing when it moves a benchmark whose axis it
|
||||
(same machine, quiet). Each benchmark prints `runs: [...]` and `mean: N ms`;
|
||||
compare the means. A pass is worth landing when it moves a benchmark whose axis it
|
||||
targets, even if the ray tracer stays flat.
|
||||
|
|
|
|||
|
|
@ -4,8 +4,7 @@
|
|||
;; targets and the ray tracer never exercises (~7% alloc).
|
||||
;;
|
||||
;; Portable Clojure: runs on jolt and JVM Clojure for cross-impl comparison.
|
||||
;; jolt -m binary-trees 14 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
|
||||
;; clojure -M -m binary-trees 14
|
||||
;; bench/run.sh binary-trees 14
|
||||
(ns binary-trees)
|
||||
|
||||
(defrecord Node [left right])
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
;; records, no collections in the hot loop) doesn't touch.
|
||||
;;
|
||||
;; Portable Clojure (jolt + JVM Clojure).
|
||||
;; jolt -m collections 200000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
|
||||
;; bench/run.sh collections 200000
|
||||
(ns collections)
|
||||
|
||||
;; map churn: accumulate a frequency map over a stream of keys, then sum it back
|
||||
|
|
|
|||
1
bench/deps.edn
Normal file
1
bench/deps.edn
Normal file
|
|
@ -0,0 +1 @@
|
|||
{:paths ["."]}
|
||||
|
|
@ -6,7 +6,7 @@
|
|||
;; float-math cost (devirt measured FLAT there).
|
||||
;;
|
||||
;; Portable Clojure (jolt + JVM Clojure).
|
||||
;; jolt -m dispatch 20000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
|
||||
;; bench/run.sh dispatch 20000
|
||||
(ns dispatch)
|
||||
|
||||
(defprotocol Shape
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
;; single-call-site / small-fn inlining and self-call direct-linking.
|
||||
;;
|
||||
;; Portable Clojure (jolt + JVM Clojure).
|
||||
;; jolt -m fib 32 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
|
||||
;; bench/run.sh fib 32
|
||||
(ns fib)
|
||||
|
||||
(defn fib [n]
|
||||
|
|
|
|||
|
|
@ -5,10 +5,10 @@
|
|||
;; on (where devirt/alloc passes measured flat), so it tracks native-arith codegen
|
||||
;; and loop quality directly.
|
||||
;;
|
||||
;; Portable Clojure (jolt + JVM Clojure).
|
||||
;; jolt -m mandelbrot 1000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
|
||||
(ns mandelbrot
|
||||
(:require [jolt.png :as png]))
|
||||
;; Portable Clojure (jolt + JVM Clojure). The jolt.png picture demo lives in
|
||||
;; mandelbrot_png.clj so this file stays portable for the JVM reference run.
|
||||
;; bench/run.sh mandelbrot 1000
|
||||
(ns mandelbrot)
|
||||
|
||||
(defn count-point [cr ci cap]
|
||||
(loop [i 0 zr 0.0 zi 0.0]
|
||||
|
|
@ -32,35 +32,6 @@
|
|||
(recur (inc y) (+ acc row)))
|
||||
acc))))
|
||||
|
||||
;; --- PNG demo (jolt.png) --------------------------------------------------
|
||||
;; Render a real picture of the set, reusing count-point as the kernel. `render`
|
||||
;; is a separate -main subcommand so the numeric-arg bench path is untouched.
|
||||
|
||||
(defn- color
|
||||
"Escape-iteration count -> RGB. In-set points (n>=cap) are black; faster
|
||||
escapes run through a warm gradient."
|
||||
[n cap]
|
||||
(if (>= n cap)
|
||||
[0 0 0]
|
||||
(let [t (/ (double n) cap)]
|
||||
[(int (* 255 (min 1.0 (* 3.0 t))))
|
||||
(int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.33))))))
|
||||
(int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.66))))))])))
|
||||
|
||||
(defn render!
|
||||
"Render a size×size view of the Mandelbrot set to a PNG at path."
|
||||
[path size]
|
||||
(let [w size h size cap 1000
|
||||
img (png/image w h)]
|
||||
(doseq [py (range h)]
|
||||
(doseq [px (range w)]
|
||||
(let [cr (- (* 3.5 (/ (double px) w)) 2.5) ; real ∈ [-2.5, 1.0]
|
||||
ci (- (* 2.8 (/ (double py) h)) 1.4) ; imag ∈ [-1.4, 1.4]
|
||||
[r g b] (color (count-point cr ci cap) cap)]
|
||||
(png/put! img r g b))))
|
||||
(png/write img w h path)
|
||||
(println "wrote" path (str w "×" h ", cap " cap))))
|
||||
|
||||
(defn- run-bench [args]
|
||||
(let [n (if (seq args) (Integer/parseInt (first args)) 1000)]
|
||||
(dotimes [_ 2] (run (quot n 2))) ; warmup
|
||||
|
|
@ -78,7 +49,4 @@
|
|||
(println "mean:" (/ (Math/round (* mean 10.0)) 10.0) "ms"))))
|
||||
|
||||
(defn -main [& args]
|
||||
(if (= (first args) "render")
|
||||
(render! (or (second args) "mandelbrot.png")
|
||||
(if (nth args 2 nil) (Integer/parseInt (nth args 2)) 600))
|
||||
(run-bench args)))
|
||||
(run-bench args))
|
||||
|
|
|
|||
36
bench/mandelbrot_png.clj
Normal file
36
bench/mandelbrot_png.clj
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
;; mandelbrot picture demo — renders a real image of the set to a PNG via
|
||||
;; jolt.png (FFI), reusing mandelbrot/count-point as the kernel. jolt-only (the
|
||||
;; benchmark in mandelbrot.clj stays portable for the JVM reference).
|
||||
;; joltc run -m mandelbrot-png [path] [size]
|
||||
(ns mandelbrot-png
|
||||
(:require [mandelbrot :as m]
|
||||
[jolt.png :as png]))
|
||||
|
||||
(defn- color
|
||||
"Escape-iteration count -> RGB. In-set points (n>=cap) are black; faster
|
||||
escapes run through a warm gradient."
|
||||
[n cap]
|
||||
(if (>= n cap)
|
||||
[0 0 0]
|
||||
(let [t (/ (double n) cap)]
|
||||
[(int (* 255 (min 1.0 (* 3.0 t))))
|
||||
(int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.33))))))
|
||||
(int (* 255 (min 1.0 (max 0.0 (* 3.0 (- t 0.66))))))])))
|
||||
|
||||
(defn render!
|
||||
"Render a size×size view of the Mandelbrot set to a PNG at path."
|
||||
[path size]
|
||||
(let [w size h size cap 1000
|
||||
img (png/image w h)]
|
||||
(doseq [py (range h)]
|
||||
(doseq [px (range w)]
|
||||
(let [cr (- (* 3.5 (/ (double px) w)) 2.5) ; real ∈ [-2.5, 1.0]
|
||||
ci (- (* 2.8 (/ (double py) h)) 1.4) ; imag ∈ [-1.4, 1.4]
|
||||
[r g b] (color (m/count-point cr ci cap) cap)]
|
||||
(png/put! img r g b))))
|
||||
(png/write img w h path)
|
||||
(println "wrote" path (str w "×" h ", cap " cap))))
|
||||
|
||||
(defn -main [& args]
|
||||
(render! (or (first args) "mandelbrot.png")
|
||||
(if (second args) (Integer/parseInt (second args)) 600)))
|
||||
|
|
@ -5,7 +5,7 @@
|
|||
;; monomorphic dispatch gets to a direct call. Same per-call work as `dispatch`.
|
||||
;;
|
||||
;; Portable Clojure (jolt + JVM Clojure).
|
||||
;; jolt -m mono-dispatch 20000 (JOLT_DIRECT_LINK=1 JOLT_WHOLE_PROGRAM=1)
|
||||
;; bench/run.sh mono-dispatch 20000
|
||||
(ns mono-dispatch)
|
||||
|
||||
(defprotocol Shape
|
||||
|
|
|
|||
80
bench/run.sh
80
bench/run.sh
|
|
@ -1,48 +1,70 @@
|
|||
#!/bin/sh
|
||||
# Run the jolt benchmark suite and print mean ms per benchmark.
|
||||
# Run the jolt benchmark suite against JVM Clojure and print a jolt/JVM scorecard.
|
||||
#
|
||||
# Each benchmark isolates an axis the ray tracer (float-compute-bound) doesn't
|
||||
# capture — see README.md. Run back-to-back against `main` to measure a pass's
|
||||
# impact.
|
||||
# jolt's optimizing passes (direct-linking, inlining, scalar-replace, whole-program
|
||||
# inference) fire only in an AOT BUILD — `joltc run -m` is unoptimized — so each
|
||||
# benchmark is compiled to an optimized standalone binary and timed. JVM Clojure
|
||||
# runs the same portable source for the absolute reference. Each benchmark prints
|
||||
# `runs: [...]` and `mean: N ms`; the table shows the means and the jolt/JVM ratio.
|
||||
#
|
||||
# bench/run.sh # default sizes, whole-program optimization on
|
||||
# JOLT_WHOLE_PROGRAM=0 bench/run.sh # compare with WP off
|
||||
# bench/run.sh binary-trees # one benchmark
|
||||
# bench/run.sh # full suite + JVM scorecard
|
||||
# bench/run.sh fib # one benchmark, default size
|
||||
# bench/run.sh fib 32 # one benchmark, custom size
|
||||
# NO_JVM=1 bench/run.sh # jolt only (skip the JVM reference)
|
||||
#
|
||||
# Needs `jolt` on PATH (build with `jpm build`; export PATH="$PWD/build:$PATH").
|
||||
# Building needs Chez's kernel dev files (libkernel.a + scheme.h) and a C compiler,
|
||||
# the same as `jolt build`; set JOLT_CHEZ_CSV to override the detected csv dir.
|
||||
set -e
|
||||
cd "$(dirname "$0")"
|
||||
root="$(cd .. && pwd)"
|
||||
joltc="$root/bin/joltc"
|
||||
export JOLT_PWD="$PWD"
|
||||
|
||||
export JOLT_DIRECT_LINK="${JOLT_DIRECT_LINK:-1}"
|
||||
export JOLT_WHOLE_PROGRAM="${JOLT_WHOLE_PROGRAM:-1}"
|
||||
export JOLT_APP_PATHS="$PWD"
|
||||
export JOLT_PATH="$PWD"
|
||||
# Locate Chez's kernel dev files for the optimized build (as build-smoke.sh does).
|
||||
csv="$JOLT_CHEZ_CSV"
|
||||
if [ -z "$csv" ]; then
|
||||
chez_bin="$(command -v chez || command -v scheme || command -v petite || true)"
|
||||
if [ -n "$chez_bin" ]; then
|
||||
base="$(cd "$(dirname "$chez_bin")/.." 2>/dev/null && pwd)"
|
||||
for d in "$base"/lib/csv*/*/; do
|
||||
[ -f "${d}libkernel.a" ] && csv="${d%/}" && break
|
||||
done
|
||||
fi
|
||||
fi
|
||||
if [ -z "$csv" ] || [ ! -f "$csv/libkernel.a" ] || [ ! -f "$csv/scheme.h" ] || ! command -v cc >/dev/null 2>&1; then
|
||||
echo "error: the optimized build needs Chez kernel dev files (libkernel.a + scheme.h) and cc." >&2
|
||||
echo " set JOLT_CHEZ_CSV to the csv dir, e.g. \$(brew --prefix chezscheme)/lib/csv*/<machine>." >&2
|
||||
exit 1
|
||||
fi
|
||||
export JOLT_CHEZ_CSV="$csv"
|
||||
|
||||
# name:default-arg (arg sized to run in a few seconds each). Axes: allocation
|
||||
# (binary-trees), megamorphic vs monomorphic dispatch, persistent-collection
|
||||
# churn (collections — now O(log n) via the HAMT, so sized up), pure
|
||||
# float compute (mandelbrot), call+arith recursion (fib).
|
||||
BENCHES="binary-trees:14 dispatch:2000 mono-dispatch:2000 collections:30000 mandelbrot:200 fib:30"
|
||||
bindir="$(mktemp -d)"
|
||||
trap 'rm -rf "$bindir"' EXIT
|
||||
|
||||
# name:default-arg, each sized to run in a few seconds. Axes: see README.md.
|
||||
BENCHES="fib:30 mandelbrot:200 collections:30000 mono-dispatch:2000 dispatch:2000 binary-trees:14"
|
||||
|
||||
# JVM=1 also runs each bench on JVM Clojure and prints a jolt/JVM ratio — the
|
||||
# holistic absolute-reference scorecard for the optimization work.
|
||||
run_one() {
|
||||
ns="${1%%:*}"; arg="${2:-${1##*:}}"
|
||||
jmean=$(jolt -m "$ns" "$arg" 2>&1 | awk '/^mean:/{print $2}')
|
||||
if [ -n "$JVM" ]; then
|
||||
vmean=$(clojure -Sdeps '{:paths ["."]}' -M -m "$ns" "$arg" 2>&1 | awk '/^mean:/{print $2}')
|
||||
ratio=$(awk "BEGIN{ if ($vmean+0>0) printf \"%.1f\", ($jmean+0)/($vmean+0); else printf \"-\" }")
|
||||
printf '%-16s jolt %9s ms jvm %8s ms %sx\n' "$ns" "${jmean:--}" "${vmean:--}" "$ratio"
|
||||
if ! "$joltc" build -m "$ns" -o "$bindir/$ns" --direct-link --opt >/dev/null 2>&1; then
|
||||
printf '%-16s jolt build FAILED\n' "$ns"; return
|
||||
fi
|
||||
jmean=$("$bindir/$ns" "$arg" 2>/dev/null | awk '/^mean:/{print $2}')
|
||||
if [ -z "$NO_JVM" ]; then
|
||||
vmean=$(clojure -Sdeps '{:paths ["."]}' -M -m "$ns" "$arg" 2>/dev/null | awk '/^mean:/{print $2}')
|
||||
ratio=$(awk "BEGIN{ if (\"$vmean\"+0>0 && \"$jmean\"+0>0) printf \"%.1fx\", (\"$jmean\"+0)/(\"$vmean\"+0); else printf \"-\" }")
|
||||
printf '%-16s jolt %9s ms jvm %8s ms %s\n' "$ns" "${jmean:--}" "${vmean:--}" "$ratio"
|
||||
else
|
||||
printf '%-16s %9s ms\n' "$ns" "${jmean:--}"
|
||||
printf '%-16s jolt %9s ms\n' "$ns" "${jmean:--}"
|
||||
fi
|
||||
}
|
||||
|
||||
if [ -n "$1" ]; then
|
||||
for spec in $BENCHES; do
|
||||
[ "${spec%%:*}" = "$1" ] && run_one "$spec" "$2"
|
||||
done
|
||||
spec=""
|
||||
for s in $BENCHES; do [ "${s%%:*}" = "$1" ] && spec="$s"; done
|
||||
[ -n "$spec" ] || { echo "unknown benchmark: $1 (have: ${BENCHES})" >&2; exit 1; }
|
||||
run_one "$spec" "$2"
|
||||
else
|
||||
echo "jolt benchmark suite (WP=$JOLT_WHOLE_PROGRAM${JVM:+, vs JVM Clojure})"
|
||||
echo "jolt benchmark suite — optimized AOT binaries${NO_JVM:+ }${NO_JVM:-, vs JVM Clojure}"
|
||||
for spec in $BENCHES; do run_one "$spec"; done
|
||||
fi
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue