jolt/bench
Dmitri Sotnikov b771908e5b
jolt.png: host PNG encoder + overlay, render mandelbrot/raytracer demos (#152)
Add a PNG writer so the demos produce actual images. Two pieces:

- src/jolt/png.janet — the encoder (8-bit RGB, filter None, stored/uncompressed
  DEFLATE so no compressor is needed; correct CRC32 + Adler32). It lives in Janet
  because per-byte work in the overlay is far too slow (a byte-array aset loop is
  ~30s for 360k bytes, and CRC32 over even a tiny image would be worse). Janet's
  bit ops are 32-bit signed, so the 32-bit-unsigned arithmetic is done with plain
  number ops (doubles hold 2^32) plus byte-level bxor. Exposed to the overlay as
  janet.png/* by importing it into eval_base's module-load-env.

- src/jolt/jolt/png.clj — the jolt.png overlay wrapper: image / put! / write. The
  overlay only produces pixels; the host encodes them in one pass.

mandelbrot gets a `render` subcommand (jolt -m mandelbrot render out.png [size])
that colours count-point's escape counts; the numeric-arg bench path is untouched.

Verified end to end: macOS `sips` accepts the output (so CRC/zlib are valid).
png-test covers the encoder structure (signature/IHDR/IEND) and the overlay
round-trip.

Co-authored-by: Yogthos <yogthos@gmail.com>
2026-06-16 21:07:10 +00:00
..
binary_trees.clj Add benchmark suite for alloc/dispatch/collection workloads (jolt-1r86) (#135) 2026-06-16 04:41:49 +00:00
collections.clj Add benchmark suite for alloc/dispatch/collection workloads (jolt-1r86) (#135) 2026-06-16 04:41:49 +00:00
dispatch.clj Add benchmark suite for alloc/dispatch/collection workloads (jolt-1r86) (#135) 2026-06-16 04:41:49 +00:00
dump-mandelbrot-emit.janet Spike: localize the mandelbrot 15x floor (jolt-5vsp) (#143) 2026-06-16 16:20:40 +00:00
fib.clj Broaden the benchmark suite; add jolt-vs-JVM scorecard (#140) 2026-06-16 14:50:38 +00:00
mandelbrot-hand-rec.janet Spike: localize the mandelbrot 15x floor (jolt-5vsp) (#143) 2026-06-16 16:20:40 +00:00
mandelbrot-hand.janet Spike: localize the mandelbrot 15x floor (jolt-5vsp) (#143) 2026-06-16 16:20:40 +00:00
mandelbrot.clj jolt.png: host PNG encoder + overlay, render mandelbrot/raytracer demos (#152) 2026-06-16 21:07:10 +00:00
mono_dispatch.clj Broaden the benchmark suite; add jolt-vs-JVM scorecard (#140) 2026-06-16 14:50:38 +00:00
README.md Broaden the benchmark suite; add jolt-vs-JVM scorecard (#140) 2026-06-16 14:50:38 +00:00
run.sh Broaden the benchmark suite; add jolt-vs-JVM scorecard (#140) 2026-06-16 14:50:38 +00:00

jolt benchmark suite

Benchmarks that isolate the workload axes jolt's optimizing passes target. The ray tracer (examples/ray-tracer) is float-compute-bound — its time is irreducible algorithmic math (hit-testing + transcendentals), and devirt, allocation removal, and type-proving all measured flat on it. So it can't tell us whether those passes work. These benchmarks make each pass's target workload the dominant cost.

Reference: the cross-language suites these draw from — Are We Fast Yet? (Marr et al., DLS '16) and the Computer Language Benchmarks Game. The benchmarks are portable Clojure, so they also run on JVM Clojure for an absolute reference.

Benchmarks

Benchmark Axis Pass it exercises Source
binary-trees allocation / GC pressure (escaping short-lived records) jolt-15jq scalar-replace, jolt-8flj escape analysis CLBG
dispatch polymorphic (megamorphic) protocol dispatch jolt-41m devirt, inline-cache AWFY-style
mono-dispatch monomorphic protocol dispatch (devirt/inline-cache can fire) jolt-41m devirt, jolt-ez5h inline-cache AWFY-style
collections persistent map/vector churn (HAMT / 32-way tries) persistent structures (jolt-684u/0hbr), transients CLBG k-nucleotide-style
mandelbrot pure float compute (tight arith loops, no alloc/dispatch) jolt-3pl native arith, loop codegen CLBG
fib recursion: function-call + integer-arith overhead jolt-3pl native arith, jolt-826 small-fn inlining CLBG

What the ray tracer does not capture and these do: allocation as the bottleneck (~7% there), megamorphic and monomorphic dispatch (its dispatch is monomorphic and cheap), persistent-collection throughput (it uses fixed records, no collections in the hot loop), and isolated compute/call overhead.

Planned additions: Richards / DeltaBlue (heavier OO dispatch), NBody (float control with record state), k-nucleotide proper.

Holistic scorecard

JVM=1 bench/run.sh runs each benchmark on jolt and JVM Clojure and prints the jolt/JVM ratio — the epic's (jolt-ffn) absolute-reference scorecard. As of the broadening (2026-06-16), ratios cluster by axis:

  • pure compute (mandelbrot) is the floor, ~15× — native arith (jolt-3pl) already gets jolt closest to the JVM.
  • collections ~28×, fib ~37×.
  • dispatch ~75× (megamorphic), and mono-dispatch is worse (~110×): the JVM inline-caches a runtime-monomorphic call site to near-free, while jolt does a full registry dispatch regardless (devirt only fires on statically proven receivers, which reduce over a vector doesn't give). This is the signal for the call-site inline cache (jolt-ez5h).
  • allocation (binary-trees) is the widest gap — but also the most inflated by host memory pressure, so read it as "alloc is the worst axis," not a precise multiple. Numbers are machine-specific; regenerate with JVM=1 bench/run.sh.

Running

jpm build && export PATH="$PWD/build:$PATH"
bench/run.sh                      # whole-program optimization on (default)
JOLT_WHOLE_PROGRAM=0 bench/run.sh # WP off, to measure what WP buys
bench/run.sh binary-trees 16      # one benchmark, custom size

A/B against a change

To measure a pass, run the suite on main, then on the branch, back to back (same machine, quiet) — the protocol used for test/bench/core-bench.janet and the ray tracer. Each benchmark prints runs: [...] and mean: N ms; compare the means. A pass is worth landing when it moves a benchmark whose axis it targets, even if the ray tracer stays flat.