jolt build: compile an app to a standalone binary (Phase 4 stages 1-2)
Restores the standalone-binary capability the Janet host had. `bin/joltc build -m NS -o OUT` AOT-compiles an app into a single self-contained executable — the whole runtime, clojure.core, stdlib and compiler embedded, no Chez install or jolt source needed at runtime. Pipeline (host/chez/build.ss, host primitive jolt.host/build-binary driven by jolt.main's build command): resolve deps, load the entry namespace recording the app namespaces in dependency order, re-emit each to Scheme, textually inline the cli.ss runtime load sequence into one flat source + the app + a launcher, then compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link against libkernel.a. Two non-obvious bits: the compile pass runs in a fresh Chez, not the loaded runtime (regex.ss shadows top-level `error`, which otherwise bakes a broken reference into the boot); and the launcher installs scheme-start rather than running -main at top level, since boot top-level forms execute during heap build before argv is set, so args only reach -main through scheme-start. Loader: a require of an in-memory namespace with no source file now no-ops, so AOT'd app namespaces satisfy require in a built binary. Mode flags (--opt/--dev, default release) are plumbed; the optimization passes they gate come in a later stage. RFC 0007 has the design. Gated by `make buildsmoke`.
This commit is contained in:
Yogthos
parent
33eff7c7d8
commit
43778eafd7
10 changed files with 554 additions and 12 deletions
186
docs/rfc/0007-compilation-modes-and-binary-output.md
Normal file
186
docs/rfc/0007-compilation-modes-and-binary-output.md
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
# RFC 0007 — Compilation modes and binary output
|
||||
|
||||
- **Status**: Draft. No code yet; this fixes the design before Phase 4 work
|
||||
(beads `jolt-cf1q.5`) starts.
|
||||
- **Champions**: jolt maintainers
|
||||
- **Created**: 2026-06-22
|
||||
|
||||
## Summary
|
||||
|
||||
Give jolt a `jolt build` command that emits a standalone executable, and a
|
||||
three-mode model that trades dynamism for speed:
|
||||
|
||||
- **dev** — open/indirect linking, redefinition works, no perf focus. What
|
||||
`repl`/`-e`/`nrepl` already are.
|
||||
- **release** (default for a built program) — direct-linked, closed-world,
|
||||
per-namespace inference. Fast, still a recognizable Clojure runtime.
|
||||
- **optimized** — whole-program inference, `fl*`/`fx*` typed emission, Chez
|
||||
whole-program optimization. Fastest, sacrifices dynamic redefinition.
|
||||
|
||||
All three already have their machinery in the tree — the inference and inline
|
||||
passes were ported into `jolt-core/jolt/passes/`. What is missing is (a) a code
|
||||
path that writes emitted Scheme to disk and AOT-compiles it instead of
|
||||
eval'ing it in process, and (b) a switch that turns the dormant passes on. This
|
||||
RFC specifies both.
|
||||
|
||||
## Motivation
|
||||
|
||||
The Janet host could produce binaries (`jolt uberscript` with dead-code
|
||||
elimination, `jolt cgen-build` for a single native binary). The Chez rehost
|
||||
dropped that machinery with the Janet host — it was Janet-specific (IR→C made
|
||||
sense when the host was Janet). On Chez the natural target is Chez's own native
|
||||
compiler, so the old emitters were deleted rather than re-pointed.
|
||||
|
||||
The result today: `bin/joltc` only ever loads the checked-in seed and
|
||||
compile-evals in process. `jolt.main/-main` dispatches `run / -M / -A / repl /
|
||||
nrepl / task` and nothing else. There is no way to ship an app as a binary, and
|
||||
the optimization passes are inert — `jolt.host/inline-enabled?` is a stub
|
||||
returning `#f` (`host/chez/host-contract.ss:283`), so every call links
|
||||
indirectly and nothing inlines. Jolt on Chez runs only in what this RFC calls
|
||||
dev mode.
|
||||
|
||||
The passes themselves survived intact:
|
||||
|
||||
- `jolt/passes/types.clj` — structural collection-type inference (RFC 0005) +
|
||||
success-type checking (RFC 0006).
|
||||
- `jolt/passes/inline.clj` — inline + flatten-lets + scalar-replace, already
|
||||
gated "direct-link only".
|
||||
- `jolt/passes/fold.clj` — const-fold, including predicate folding.
|
||||
|
||||
So this is not a port of lost code. It is wiring: a build front-end, a
|
||||
file-emitting back-end path, and a mode switch over passes that already exist.
|
||||
|
||||
## The three modes
|
||||
|
||||
| Mode | Linking | Inference | Redefinition | Driver |
|
||||
|---|---|---|---|---|
|
||||
| **dev** | indirect (var-deref per call) | off | yes | `repl`, `-e`, `nrepl`, `run` of a file by default |
|
||||
| **release** | direct, closed-world | per-namespace | no (closed world) | `jolt build` default |
|
||||
| **optimized** | direct + whole-program | whole-program fixpoint, `fl*`/`fx*` | no | `jolt build --opt` / `-M`-style entry |
|
||||
|
||||
The modes are points on one axis (how much the back end may assume is fixed),
|
||||
not three code paths. Each mode is a setting of two independent knobs the passes
|
||||
already understand:
|
||||
|
||||
- **direct-link?** — may a call to a var compile to a direct procedure
|
||||
reference instead of a `var-deref`? Enables inlining and call-site folding.
|
||||
Opt-out is per-target: a `^:redef` or `^:dynamic` var always links indirect.
|
||||
- **whole-program?** — does inference see the whole reachable program at once
|
||||
(closed world), so a record param's callers in other namespaces are visible
|
||||
and its field reads specialize? Without it, inference is per-namespace and a
|
||||
cross-ns param de-specializes to `:any` (the cross-ns penalty documented in
|
||||
the `cross-ns-param-penalty` memory; declared `^RecordType` hints are the
|
||||
open-world escape hatch).
|
||||
|
||||
```
|
||||
dev: direct-link? = false whole-program? = false
|
||||
release: direct-link? = true whole-program? = false
|
||||
optimized: direct-link? = true whole-program? = true
|
||||
```
|
||||
|
||||
`fl*`/`fx*` typed emission (unchecked flonum/fixnum Scheme ops) rides on
|
||||
optimized: only whole-program inference proves the types that make dropping the
|
||||
numeric-tower dispatch sound. Release keeps the tower.
|
||||
|
||||
## CLI surface
|
||||
|
||||
```
|
||||
jolt build [-m NS | FILE] [-o OUT] [--opt] [--dev]
|
||||
```
|
||||
|
||||
- Resolves `deps.edn` exactly as `run` does (reuse `jolt.deps`).
|
||||
- Default mode is **release**. `--opt` selects optimized; `--dev` builds an
|
||||
unoptimized binary (useful to ship a debuggable build, not for the REPL).
|
||||
- `-o` names the output (default the entry ns / file stem).
|
||||
- Output is a single executable: a Chez boot file plus the compiled program,
|
||||
launched by a thin wrapper, or a fully linked image where the platform allows.
|
||||
App libraries are baked in — no source roots needed at runtime.
|
||||
|
||||
Env opt-outs for the build (mirrors the Janet knobs, now keyed off the mode
|
||||
rather than the run): `JOLT_NO_DIRECT_LINK` forces open linking even in a build,
|
||||
`JOLT_NO_WHOLE_PROGRAM` keeps direct-link but per-namespace, `JOLT_WHOLE_PROGRAM=1`
|
||||
forces whole-program. These already name the two knobs above.
|
||||
|
||||
## Emission pipeline
|
||||
|
||||
The in-process spine today (`host/chez/compile-eval.ss`) is, per form:
|
||||
|
||||
```
|
||||
source → read → analyze (→ IR) → emit (→ Scheme string) → (eval (read …))
|
||||
```
|
||||
|
||||
`jolt build` keeps everything up to `emit` and replaces the per-form `eval` with
|
||||
accumulate-then-compile:
|
||||
|
||||
1. **Assemble the program.** Starting from the entry ns's `-main`, load the
|
||||
transitive `require` graph (the loader already does this) and collect every
|
||||
reachable top-level form, in dependency order, with its compile namespace.
|
||||
2. **Dead-code elimination.** Re-target the uberscript DCE idea: compute
|
||||
reachability from `-main` plus non-prunable forms, drop dead `defn`/`defn-`.
|
||||
Bail to keep-all on `resolve`/`ns-resolve`/`requiring-resolve`/`find-var`/
|
||||
`intern`/`eval`/`load-string` (anything that defeats static reachability);
|
||||
keep and scan `defmethod`/`defrecord`/`extend` bodies so dispatch targets
|
||||
stay live.
|
||||
3. **Emit to a file.** Run `analyze → emit` for each surviving form under the
|
||||
mode's knobs, concatenating the Scheme strings into one program source (the
|
||||
core overlay prelude first, exactly as the seed image is built today).
|
||||
4. **Compile.** Feed that source to Chez `compile-program` (release) or
|
||||
`compile-whole-program` (optimized, which also lets Chez cross-module
|
||||
inline), producing a compiled object, then link a boot file / wrapper into
|
||||
the final executable.
|
||||
|
||||
Steps 3–4 are the only genuinely new back-end code. Step 2 is a re-port of a
|
||||
deleted pass. Steps before them already run on every `joltc` invocation.
|
||||
|
||||
## Turning the passes on
|
||||
|
||||
`inline-enabled?` is the existing gate. Today `host-contract.ss` hardwires it to
|
||||
`#f`. Under this RFC the build sets it (and a parallel `whole-program?` flag)
|
||||
from the chosen mode before compiling, so:
|
||||
|
||||
- release: `inline-enabled?` → true, whole-program off. Per-ns inference and
|
||||
inlining light up; `fl*`/`fx*` stays off.
|
||||
- optimized: both on; the types pass runs its whole-program fixpoint and the
|
||||
back end may emit unchecked numeric ops where a flonum/fixnum is proven.
|
||||
|
||||
No new pass is required to reach release — it is the ported passes, ungated.
|
||||
|
||||
## Staging
|
||||
|
||||
1. **Spike (de-risk Chez AOT).** Emit a trivial whole program to disk and prove
|
||||
`compile-program` + boot/static link yields a standalone binary that runs.
|
||||
This is the only real unknown.
|
||||
2. **`jolt build` release.** Front-end + file-emitting back-end path + flip
|
||||
`inline-enabled?` from the mode. Gate against the bench/corpus suites; binary
|
||||
output must pass the corpus a `run` passes.
|
||||
3. **DCE.** Re-port the reachability pass; gate with a test like the old
|
||||
`uberscript-dce` case.
|
||||
4. **Optimized.** Whole-program flag, `compile-whole-program`, `fl*`/`fx*`
|
||||
emission. Gate on the bench suite (ray tracer, binary-trees) for size and
|
||||
speed vs the spike baseline.
|
||||
|
||||
Each stage is TDD against the existing gates (`make test`, `make corpus`, the
|
||||
`bench/` programs). Modes land behind the build command, so dev — the only mode
|
||||
today — is unaffected until a stage proves out.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Static vs. boot-file linking.** A fully static Chez image is the smallest,
|
||||
most portable artifact but the most work to link; a boot file plus a stub
|
||||
launcher is the easy first cut. Spike decides which step 1 targets.
|
||||
- **FFI in a built binary.** `jolt.ffi` loads native libraries at runtime; a
|
||||
closed-world build still needs that to work. The build must bake the FFI
|
||||
Clojure side and keep dynamic `dlopen` at run time.
|
||||
- **Macro and `eval` at runtime.** Release/optimized are closed-world, but an
|
||||
app that calls `eval`/`load-string` needs the compiler present. Either ship
|
||||
the compiler image in the binary (larger) or reject those builds (the DCE
|
||||
bail-out already detects the calls).
|
||||
|
||||
## Prior art in this repo
|
||||
|
||||
The optimization design these modes turn on is RFC 0004 (type hints), RFC 0005
|
||||
(structural inference), RFC 0006 (success checking). The linking model — direct
|
||||
linking as a per-unit property, `^:redef`/`^:dynamic` as the only opt-out — and
|
||||
the cross-ns specialization penalty are recorded in beads memories
|
||||
(`jolt-linking-model`, `cross-ns-param-penalty`). Phase 4 (`jolt-cf1q.5`) is the
|
||||
tracking issue.
|
||||
Loading…
Add table
Add a link
Reference in a new issue