Broaden the benchmark suite; add jolt-vs-JVM scorecard (#140)

The ray tracer is compute-bound and the three existing benches only cover
alloc / megamorphic-dispatch / collections. Add three axes the epic needs to
judge itself holistically:

- mono-dispatch: monomorphic protocol dispatch. Its jolt/JVM ratio (~110x) is
  *worse* than megamorphic (~76x) — the JVM inline-caches a runtime-monomorphic
  call site to near-free while jolt does a full registry dispatch (devirt only
  fires on statically-proven receivers). Points at the call-site inline cache.
- mandelbrot: pure float compute, no alloc/dispatch. The floor at ~15x — native
  arith already gets close to the JVM.
- fib: recursion, call + integer-arith overhead.

run.sh gains JVM=1, which runs each bench on JVM Clojure too and prints the
jolt/JVM ratio. collections sized up now that the map is a HAMT (jolt-684u).
README documents the axes and the current scorecard.

Co-authored-by: Yogthos <yogthos@gmail.com>
This commit is contained in:
Dmitri Sotnikov 2026-06-16 14:50:38 +00:00 committed by GitHub
parent f0293fb4ee
commit 7d0b1d5695
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 166 additions and 15 deletions

View file

@ -18,17 +18,37 @@ absolute reference.
| Benchmark | Axis | Pass it exercises | Source |
|---|---|---|---|
| `binary-trees` | allocation / GC pressure (escaping short-lived records) | jolt-15jq scalar-replace, jolt-8flj escape analysis | CLBG |
| `dispatch` | polymorphic (megamorphic) protocol dispatch | jolt-41m devirt, inline-cache | AWFY-style |
| `collections` | persistent map/vector churn (32-way tries) | persistent structures, transients | CLBG k-nucleotide-style |
| `dispatch` | polymorphic (**megamorphic**) protocol dispatch | jolt-41m devirt, inline-cache | AWFY-style |
| `mono-dispatch` | **monomorphic** protocol dispatch (devirt/inline-cache *can* fire) | jolt-41m devirt, jolt-ez5h inline-cache | AWFY-style |
| `collections` | persistent map/vector churn (HAMT / 32-way tries) | persistent structures (jolt-684u/0hbr), transients | CLBG k-nucleotide-style |
| `mandelbrot` | pure float compute (tight arith loops, no alloc/dispatch) | jolt-3pl native arith, loop codegen | CLBG |
| `fib` | recursion: function-call + integer-arith overhead | jolt-3pl native arith, jolt-826 small-fn inlining | CLBG |
What the ray tracer does **not** capture and these do: allocation as the
bottleneck (~7% there), megamorphic dispatch (its dispatch is monomorphic and
cheap), and persistent-collection throughput (it uses fixed records, no
collections in the hot loop).
bottleneck (~7% there), megamorphic *and* monomorphic dispatch (its dispatch is
monomorphic and cheap), persistent-collection throughput (it uses fixed records,
no collections in the hot loop), and isolated compute/call overhead.
Planned additions: Richards / DeltaBlue (heavier OO dispatch), a **monomorphic**
dispatch variant (where devirt *can* fire — the megamorphic `dispatch` can't),
NBody (float control, parallels the ray tracer), k-nucleotide proper.
Planned additions: Richards / DeltaBlue (heavier OO dispatch), NBody (float
control with record state), k-nucleotide proper.
## Holistic scorecard
`JVM=1 bench/run.sh` runs each benchmark on jolt **and** JVM Clojure and prints
the jolt/JVM ratio — the epic's (jolt-ffn) absolute-reference scorecard. As of
the broadening (2026-06-16), ratios cluster by axis:
- **pure compute** (`mandelbrot`) is the floor, ~15× — native arith (jolt-3pl)
already gets jolt closest to the JVM.
- **collections** ~28×, **fib** ~37×.
- **dispatch** ~75× (megamorphic), and `mono-dispatch` is *worse* (~110×): the
JVM inline-caches a runtime-monomorphic call site to near-free, while jolt does
a full registry dispatch regardless (devirt only fires on *statically* proven
receivers, which `reduce` over a vector doesn't give). This is the signal for
the call-site inline cache (jolt-ez5h).
- **allocation** (`binary-trees`) is the widest gap — but also the most inflated
by host memory pressure, so read it as "alloc is the worst axis," not a precise
multiple. Numbers are machine-specific; regenerate with `JVM=1 bench/run.sh`.
## Running