43778eafd7
Restores the standalone-binary capability the Janet host had. `bin/joltc build -m NS -o OUT` AOT-compiles an app into a single self-contained executable — the whole runtime, clojure.core, stdlib and compiler embedded, no Chez install or jolt source needed at runtime. Pipeline (host/chez/build.ss, host primitive jolt.host/build-binary driven by jolt.main's build command): resolve deps, load the entry namespace recording the app namespaces in dependency order, re-emit each to Scheme, textually inline the cli.ss runtime load sequence into one flat source + the app + a launcher, then compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link against libkernel.a. Two non-obvious bits: the compile pass runs in a fresh Chez, not the loaded runtime (regex.ss shadows top-level `error`, which otherwise bakes a broken reference into the boot); and the launcher installs scheme-start rather than running -main at top level, since boot top-level forms execute during heap build before argv is set, so args only reach -main through scheme-start. Loader: a require of an in-memory namespace with no source file now no-ops, so AOT'd app namespaces satisfy require in a built binary. Mode flags (--opt/--dev, default release) are plumbed; the optimization passes they gate come in a later stage. RFC 0007 has the design. Gated by `make buildsmoke`.
186 lines
9 KiB
Markdown
186 lines
9 KiB
Markdown
# RFC 0007 — Compilation modes and binary output
|
||
|
||
- **Status**: Draft. No code yet; this fixes the design before Phase 4 work
|
||
(beads `jolt-cf1q.5`) starts.
|
||
- **Champions**: jolt maintainers
|
||
- **Created**: 2026-06-22
|
||
|
||
## Summary
|
||
|
||
Give jolt a `jolt build` command that emits a standalone executable, and a
|
||
three-mode model that trades dynamism for speed:
|
||
|
||
- **dev** — open/indirect linking, redefinition works, no perf focus. What
|
||
`repl`/`-e`/`nrepl` already are.
|
||
- **release** (default for a built program) — direct-linked, closed-world,
|
||
per-namespace inference. Fast, still a recognizable Clojure runtime.
|
||
- **optimized** — whole-program inference, `fl*`/`fx*` typed emission, Chez
|
||
whole-program optimization. Fastest, sacrifices dynamic redefinition.
|
||
|
||
All three already have their machinery in the tree — the inference and inline
|
||
passes were ported into `jolt-core/jolt/passes/`. What is missing is (a) a code
|
||
path that writes emitted Scheme to disk and AOT-compiles it instead of
|
||
eval'ing it in process, and (b) a switch that turns the dormant passes on. This
|
||
RFC specifies both.
|
||
|
||
## Motivation
|
||
|
||
The Janet host could produce binaries (`jolt uberscript` with dead-code
|
||
elimination, `jolt cgen-build` for a single native binary). The Chez rehost
|
||
dropped that machinery with the Janet host — it was Janet-specific (IR→C made
|
||
sense when the host was Janet). On Chez the natural target is Chez's own native
|
||
compiler, so the old emitters were deleted rather than re-pointed.
|
||
|
||
The result today: `bin/joltc` only ever loads the checked-in seed and
|
||
compile-evals in process. `jolt.main/-main` dispatches `run / -M / -A / repl /
|
||
nrepl / task` and nothing else. There is no way to ship an app as a binary, and
|
||
the optimization passes are inert — `jolt.host/inline-enabled?` is a stub
|
||
returning `#f` (`host/chez/host-contract.ss:283`), so every call links
|
||
indirectly and nothing inlines. Jolt on Chez runs only in what this RFC calls
|
||
dev mode.
|
||
|
||
The passes themselves survived intact:
|
||
|
||
- `jolt/passes/types.clj` — structural collection-type inference (RFC 0005) +
|
||
success-type checking (RFC 0006).
|
||
- `jolt/passes/inline.clj` — inline + flatten-lets + scalar-replace, already
|
||
gated "direct-link only".
|
||
- `jolt/passes/fold.clj` — const-fold, including predicate folding.
|
||
|
||
So this is not a port of lost code. It is wiring: a build front-end, a
|
||
file-emitting back-end path, and a mode switch over passes that already exist.
|
||
|
||
## The three modes
|
||
|
||
| Mode | Linking | Inference | Redefinition | Driver |
|
||
|---|---|---|---|---|
|
||
| **dev** | indirect (var-deref per call) | off | yes | `repl`, `-e`, `nrepl`, `run` of a file by default |
|
||
| **release** | direct, closed-world | per-namespace | no (closed world) | `jolt build` default |
|
||
| **optimized** | direct + whole-program | whole-program fixpoint, `fl*`/`fx*` | no | `jolt build --opt` / `-M`-style entry |
|
||
|
||
The modes are points on one axis (how much the back end may assume is fixed),
|
||
not three code paths. Each mode is a setting of two independent knobs the passes
|
||
already understand:
|
||
|
||
- **direct-link?** — may a call to a var compile to a direct procedure
|
||
reference instead of a `var-deref`? Enables inlining and call-site folding.
|
||
Opt-out is per-target: a `^:redef` or `^:dynamic` var always links indirect.
|
||
- **whole-program?** — does inference see the whole reachable program at once
|
||
(closed world), so a record param's callers in other namespaces are visible
|
||
and its field reads specialize? Without it, inference is per-namespace and a
|
||
cross-ns param de-specializes to `:any` (the cross-ns penalty documented in
|
||
the `cross-ns-param-penalty` memory; declared `^RecordType` hints are the
|
||
open-world escape hatch).
|
||
|
||
```
|
||
dev: direct-link? = false whole-program? = false
|
||
release: direct-link? = true whole-program? = false
|
||
optimized: direct-link? = true whole-program? = true
|
||
```
|
||
|
||
`fl*`/`fx*` typed emission (unchecked flonum/fixnum Scheme ops) rides on
|
||
optimized: only whole-program inference proves the types that make dropping the
|
||
numeric-tower dispatch sound. Release keeps the tower.
|
||
|
||
## CLI surface
|
||
|
||
```
|
||
jolt build [-m NS | FILE] [-o OUT] [--opt] [--dev]
|
||
```
|
||
|
||
- Resolves `deps.edn` exactly as `run` does (reuse `jolt.deps`).
|
||
- Default mode is **release**. `--opt` selects optimized; `--dev` builds an
|
||
unoptimized binary (useful to ship a debuggable build, not for the REPL).
|
||
- `-o` names the output (default the entry ns / file stem).
|
||
- Output is a single executable: a Chez boot file plus the compiled program,
|
||
launched by a thin wrapper, or a fully linked image where the platform allows.
|
||
App libraries are baked in — no source roots needed at runtime.
|
||
|
||
Env opt-outs for the build (mirrors the Janet knobs, now keyed off the mode
|
||
rather than the run): `JOLT_NO_DIRECT_LINK` forces open linking even in a build,
|
||
`JOLT_NO_WHOLE_PROGRAM` keeps direct-link but per-namespace, `JOLT_WHOLE_PROGRAM=1`
|
||
forces whole-program. These already name the two knobs above.
|
||
|
||
## Emission pipeline
|
||
|
||
The in-process spine today (`host/chez/compile-eval.ss`) is, per form:
|
||
|
||
```
|
||
source → read → analyze (→ IR) → emit (→ Scheme string) → (eval (read …))
|
||
```
|
||
|
||
`jolt build` keeps everything up to `emit` and replaces the per-form `eval` with
|
||
accumulate-then-compile:
|
||
|
||
1. **Assemble the program.** Starting from the entry ns's `-main`, load the
|
||
transitive `require` graph (the loader already does this) and collect every
|
||
reachable top-level form, in dependency order, with its compile namespace.
|
||
2. **Dead-code elimination.** Re-target the uberscript DCE idea: compute
|
||
reachability from `-main` plus non-prunable forms, drop dead `defn`/`defn-`.
|
||
Bail to keep-all on `resolve`/`ns-resolve`/`requiring-resolve`/`find-var`/
|
||
`intern`/`eval`/`load-string` (anything that defeats static reachability);
|
||
keep and scan `defmethod`/`defrecord`/`extend` bodies so dispatch targets
|
||
stay live.
|
||
3. **Emit to a file.** Run `analyze → emit` for each surviving form under the
|
||
mode's knobs, concatenating the Scheme strings into one program source (the
|
||
core overlay prelude first, exactly as the seed image is built today).
|
||
4. **Compile.** Feed that source to Chez `compile-program` (release) or
|
||
`compile-whole-program` (optimized, which also lets Chez cross-module
|
||
inline), producing a compiled object, then link a boot file / wrapper into
|
||
the final executable.
|
||
|
||
Steps 3–4 are the only genuinely new back-end code. Step 2 is a re-port of a
|
||
deleted pass. Steps before them already run on every `joltc` invocation.
|
||
|
||
## Turning the passes on
|
||
|
||
`inline-enabled?` is the existing gate. Today `host-contract.ss` hardwires it to
|
||
`#f`. Under this RFC the build sets it (and a parallel `whole-program?` flag)
|
||
from the chosen mode before compiling, so:
|
||
|
||
- release: `inline-enabled?` → true, whole-program off. Per-ns inference and
|
||
inlining light up; `fl*`/`fx*` stays off.
|
||
- optimized: both on; the types pass runs its whole-program fixpoint and the
|
||
back end may emit unchecked numeric ops where a flonum/fixnum is proven.
|
||
|
||
No new pass is required to reach release — it is the ported passes, ungated.
|
||
|
||
## Staging
|
||
|
||
1. **Spike (de-risk Chez AOT).** Emit a trivial whole program to disk and prove
|
||
`compile-program` + boot/static link yields a standalone binary that runs.
|
||
This is the only real unknown.
|
||
2. **`jolt build` release.** Front-end + file-emitting back-end path + flip
|
||
`inline-enabled?` from the mode. Gate against the bench/corpus suites; binary
|
||
output must pass the corpus a `run` passes.
|
||
3. **DCE.** Re-port the reachability pass; gate with a test like the old
|
||
`uberscript-dce` case.
|
||
4. **Optimized.** Whole-program flag, `compile-whole-program`, `fl*`/`fx*`
|
||
emission. Gate on the bench suite (ray tracer, binary-trees) for size and
|
||
speed vs the spike baseline.
|
||
|
||
Each stage is TDD against the existing gates (`make test`, `make corpus`, the
|
||
`bench/` programs). Modes land behind the build command, so dev — the only mode
|
||
today — is unaffected until a stage proves out.
|
||
|
||
## Open questions
|
||
|
||
- **Static vs. boot-file linking.** A fully static Chez image is the smallest,
|
||
most portable artifact but the most work to link; a boot file plus a stub
|
||
launcher is the easy first cut. Spike decides which step 1 targets.
|
||
- **FFI in a built binary.** `jolt.ffi` loads native libraries at runtime; a
|
||
closed-world build still needs that to work. The build must bake the FFI
|
||
Clojure side and keep dynamic `dlopen` at run time.
|
||
- **Macro and `eval` at runtime.** Release/optimized are closed-world, but an
|
||
app that calls `eval`/`load-string` needs the compiler present. Either ship
|
||
the compiler image in the binary (larger) or reject those builds (the DCE
|
||
bail-out already detects the calls).
|
||
|
||
## Prior art in this repo
|
||
|
||
The optimization design these modes turn on is RFC 0004 (type hints), RFC 0005
|
||
(structural inference), RFC 0006 (success checking). The linking model — direct
|
||
linking as a per-unit property, `^:redef`/`^:dynamic` as the only opt-out — and
|
||
the cross-ns specialization penalty are recorded in beads memories
|
||
(`jolt-linking-model`, `cross-ns-param-penalty`). Phase 4 (`jolt-cf1q.5`) is the
|
||
tracking issue.
|