jolt build: compile an app to a standalone binary (Phase 4 stages 1-2)
Restores the standalone-binary capability the Janet host had. `bin/joltc build -m NS -o OUT` AOT-compiles an app into a single self-contained executable — the whole runtime, clojure.core, stdlib and compiler embedded, no Chez install or jolt source needed at runtime. Pipeline (host/chez/build.ss, host primitive jolt.host/build-binary driven by jolt.main's build command): resolve deps, load the entry namespace recording the app namespaces in dependency order, re-emit each to Scheme, textually inline the cli.ss runtime load sequence into one flat source + the app + a launcher, then compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link against libkernel.a. Two non-obvious bits: the compile pass runs in a fresh Chez, not the loaded runtime (regex.ss shadows top-level `error`, which otherwise bakes a broken reference into the boot); and the launcher installs scheme-start rather than running -main at top level, since boot top-level forms execute during heap build before argv is set, so args only reach -main through scheme-start. Loader: a require of an in-memory namespace with no source file now no-ops, so AOT'd app namespaces satisfy require in a built binary. Mode flags (--opt/--dev, default release) are plumbed; the optimization passes they gate come in a later stage. RFC 0007 has the design. Gated by `make buildsmoke`.
This commit is contained in:
Yogthos
parent
33eff7c7d8
commit
43778eafd7
10 changed files with 554 additions and 12 deletions
8
Makefile
8
Makefile
|
|
@ -4,7 +4,7 @@
|
||||||
# build step. `make test` is the full gate. `make remint` rebuilds the seed after a
|
# build step. `make test` is the full gate. `make remint` rebuilds the seed after a
|
||||||
# source change.
|
# source change.
|
||||||
|
|
||||||
.PHONY: test ci values corpus unit smoke selfhost sci certify ffi transient remint
|
.PHONY: test ci values corpus unit smoke buildsmoke selfhost sci certify ffi transient remint
|
||||||
|
|
||||||
# Full gate (dev machine). Includes the self-host byte-fixpoint, which only holds
|
# Full gate (dev machine). Includes the self-host byte-fixpoint, which only holds
|
||||||
# on the same Chez that minted the seed.
|
# on the same Chez that minted the seed.
|
||||||
|
|
@ -15,7 +15,7 @@ test: selfhost ci
|
||||||
# lockfile) — it RUNS correctly on any Chez, but `selfhost` rebuilds it and a
|
# lockfile) — it RUNS correctly on any Chez, but `selfhost` rebuilds it and a
|
||||||
# different Chez version may emit byte-different (gensym/order) output, so the
|
# different Chez version may emit byte-different (gensym/order) output, so the
|
||||||
# byte-fixpoint is a dev-machine check, not a CI one (jolt-8479).
|
# byte-fixpoint is a dev-machine check, not a CI one (jolt-8479).
|
||||||
ci: values corpus unit smoke sci ffi transient certify
|
ci: values corpus unit smoke buildsmoke sci ffi transient certify
|
||||||
@echo "OK: CI gates passed"
|
@echo "OK: CI gates passed"
|
||||||
|
|
||||||
# Self-host fixpoint: bootstrap.ss rebuild == checked-in seed.
|
# Self-host fixpoint: bootstrap.ss rebuild == checked-in seed.
|
||||||
|
|
@ -38,6 +38,10 @@ unit:
|
||||||
smoke:
|
smoke:
|
||||||
@sh host/chez/smoke.sh
|
@sh host/chez/smoke.sh
|
||||||
|
|
||||||
|
# `jolt build` produces a working standalone binary.
|
||||||
|
buildsmoke:
|
||||||
|
@sh host/chez/build-smoke.sh
|
||||||
|
|
||||||
# SCI conformance: load borkdude/sci's source through joltc (floor-gated).
|
# SCI conformance: load borkdude/sci's source through joltc (floor-gated).
|
||||||
sci:
|
sci:
|
||||||
@chez --script host/chez/run-sci.ss
|
@chez --script host/chez/run-sci.ss
|
||||||
|
|
|
||||||
186
docs/rfc/0007-compilation-modes-and-binary-output.md
Normal file
186
docs/rfc/0007-compilation-modes-and-binary-output.md
Normal file
|
|
@ -0,0 +1,186 @@
|
||||||
|
# RFC 0007 — Compilation modes and binary output
|
||||||
|
|
||||||
|
- **Status**: Draft. No code yet; this fixes the design before Phase 4 work
|
||||||
|
(beads `jolt-cf1q.5`) starts.
|
||||||
|
- **Champions**: jolt maintainers
|
||||||
|
- **Created**: 2026-06-22
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Give jolt a `jolt build` command that emits a standalone executable, and a
|
||||||
|
three-mode model that trades dynamism for speed:
|
||||||
|
|
||||||
|
- **dev** — open/indirect linking, redefinition works, no perf focus. What
|
||||||
|
`repl`/`-e`/`nrepl` already are.
|
||||||
|
- **release** (default for a built program) — direct-linked, closed-world,
|
||||||
|
per-namespace inference. Fast, still a recognizable Clojure runtime.
|
||||||
|
- **optimized** — whole-program inference, `fl*`/`fx*` typed emission, Chez
|
||||||
|
whole-program optimization. Fastest, sacrifices dynamic redefinition.
|
||||||
|
|
||||||
|
All three already have their machinery in the tree — the inference and inline
|
||||||
|
passes were ported into `jolt-core/jolt/passes/`. What is missing is (a) a code
|
||||||
|
path that writes emitted Scheme to disk and AOT-compiles it instead of
|
||||||
|
eval'ing it in process, and (b) a switch that turns the dormant passes on. This
|
||||||
|
RFC specifies both.
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
The Janet host could produce binaries (`jolt uberscript` with dead-code
|
||||||
|
elimination, `jolt cgen-build` for a single native binary). The Chez rehost
|
||||||
|
dropped that machinery with the Janet host — it was Janet-specific (IR→C made
|
||||||
|
sense when the host was Janet). On Chez the natural target is Chez's own native
|
||||||
|
compiler, so the old emitters were deleted rather than re-pointed.
|
||||||
|
|
||||||
|
The result today: `bin/joltc` only ever loads the checked-in seed and
|
||||||
|
compile-evals in process. `jolt.main/-main` dispatches `run / -M / -A / repl /
|
||||||
|
nrepl / task` and nothing else. There is no way to ship an app as a binary, and
|
||||||
|
the optimization passes are inert — `jolt.host/inline-enabled?` is a stub
|
||||||
|
returning `#f` (`host/chez/host-contract.ss:283`), so every call links
|
||||||
|
indirectly and nothing inlines. Jolt on Chez runs only in what this RFC calls
|
||||||
|
dev mode.
|
||||||
|
|
||||||
|
The passes themselves survived intact:
|
||||||
|
|
||||||
|
- `jolt/passes/types.clj` — structural collection-type inference (RFC 0005) +
|
||||||
|
success-type checking (RFC 0006).
|
||||||
|
- `jolt/passes/inline.clj` — inline + flatten-lets + scalar-replace, already
|
||||||
|
gated "direct-link only".
|
||||||
|
- `jolt/passes/fold.clj` — const-fold, including predicate folding.
|
||||||
|
|
||||||
|
So this is not a port of lost code. It is wiring: a build front-end, a
|
||||||
|
file-emitting back-end path, and a mode switch over passes that already exist.
|
||||||
|
|
||||||
|
## The three modes
|
||||||
|
|
||||||
|
| Mode | Linking | Inference | Redefinition | Driver |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **dev** | indirect (var-deref per call) | off | yes | `repl`, `-e`, `nrepl`, `run` of a file by default |
|
||||||
|
| **release** | direct, closed-world | per-namespace | no (closed world) | `jolt build` default |
|
||||||
|
| **optimized** | direct + whole-program | whole-program fixpoint, `fl*`/`fx*` | no | `jolt build --opt` / `-M`-style entry |
|
||||||
|
|
||||||
|
The modes are points on one axis (how much the back end may assume is fixed),
|
||||||
|
not three code paths. Each mode is a setting of two independent knobs the passes
|
||||||
|
already understand:
|
||||||
|
|
||||||
|
- **direct-link?** — may a call to a var compile to a direct procedure
|
||||||
|
reference instead of a `var-deref`? Enables inlining and call-site folding.
|
||||||
|
Opt-out is per-target: a `^:redef` or `^:dynamic` var always links indirect.
|
||||||
|
- **whole-program?** — does inference see the whole reachable program at once
|
||||||
|
(closed world), so a record param's callers in other namespaces are visible
|
||||||
|
and its field reads specialize? Without it, inference is per-namespace and a
|
||||||
|
cross-ns param de-specializes to `:any` (the cross-ns penalty documented in
|
||||||
|
the `cross-ns-param-penalty` memory; declared `^RecordType` hints are the
|
||||||
|
open-world escape hatch).
|
||||||
|
|
||||||
|
```
|
||||||
|
dev: direct-link? = false whole-program? = false
|
||||||
|
release: direct-link? = true whole-program? = false
|
||||||
|
optimized: direct-link? = true whole-program? = true
|
||||||
|
```
|
||||||
|
|
||||||
|
`fl*`/`fx*` typed emission (unchecked flonum/fixnum Scheme ops) rides on
|
||||||
|
optimized: only whole-program inference proves the types that make dropping the
|
||||||
|
numeric-tower dispatch sound. Release keeps the tower.
|
||||||
|
|
||||||
|
## CLI surface
|
||||||
|
|
||||||
|
```
|
||||||
|
jolt build [-m NS | FILE] [-o OUT] [--opt] [--dev]
|
||||||
|
```
|
||||||
|
|
||||||
|
- Resolves `deps.edn` exactly as `run` does (reuse `jolt.deps`).
|
||||||
|
- Default mode is **release**. `--opt` selects optimized; `--dev` builds an
|
||||||
|
unoptimized binary (useful to ship a debuggable build, not for the REPL).
|
||||||
|
- `-o` names the output (default the entry ns / file stem).
|
||||||
|
- Output is a single executable: a Chez boot file plus the compiled program,
|
||||||
|
launched by a thin wrapper, or a fully linked image where the platform allows.
|
||||||
|
App libraries are baked in — no source roots needed at runtime.
|
||||||
|
|
||||||
|
Env opt-outs for the build (mirrors the Janet knobs, now keyed off the mode
|
||||||
|
rather than the run): `JOLT_NO_DIRECT_LINK` forces open linking even in a build,
|
||||||
|
`JOLT_NO_WHOLE_PROGRAM` keeps direct-link but per-namespace, `JOLT_WHOLE_PROGRAM=1`
|
||||||
|
forces whole-program. These already name the two knobs above.
|
||||||
|
|
||||||
|
## Emission pipeline
|
||||||
|
|
||||||
|
The in-process spine today (`host/chez/compile-eval.ss`) is, per form:
|
||||||
|
|
||||||
|
```
|
||||||
|
source → read → analyze (→ IR) → emit (→ Scheme string) → (eval (read …))
|
||||||
|
```
|
||||||
|
|
||||||
|
`jolt build` keeps everything up to `emit` and replaces the per-form `eval` with
|
||||||
|
accumulate-then-compile:
|
||||||
|
|
||||||
|
1. **Assemble the program.** Starting from the entry ns's `-main`, load the
|
||||||
|
transitive `require` graph (the loader already does this) and collect every
|
||||||
|
reachable top-level form, in dependency order, with its compile namespace.
|
||||||
|
2. **Dead-code elimination.** Re-target the uberscript DCE idea: compute
|
||||||
|
reachability from `-main` plus non-prunable forms, drop dead `defn`/`defn-`.
|
||||||
|
Bail to keep-all on `resolve`/`ns-resolve`/`requiring-resolve`/`find-var`/
|
||||||
|
`intern`/`eval`/`load-string` (anything that defeats static reachability);
|
||||||
|
keep and scan `defmethod`/`defrecord`/`extend` bodies so dispatch targets
|
||||||
|
stay live.
|
||||||
|
3. **Emit to a file.** Run `analyze → emit` for each surviving form under the
|
||||||
|
mode's knobs, concatenating the Scheme strings into one program source (the
|
||||||
|
core overlay prelude first, exactly as the seed image is built today).
|
||||||
|
4. **Compile.** Feed that source to Chez `compile-program` (release) or
|
||||||
|
`compile-whole-program` (optimized, which also lets Chez cross-module
|
||||||
|
inline), producing a compiled object, then link a boot file / wrapper into
|
||||||
|
the final executable.
|
||||||
|
|
||||||
|
Steps 3–4 are the only genuinely new back-end code. Step 2 is a re-port of a
|
||||||
|
deleted pass. Steps before them already run on every `joltc` invocation.
|
||||||
|
|
||||||
|
## Turning the passes on
|
||||||
|
|
||||||
|
`inline-enabled?` is the existing gate. Today `host-contract.ss` hardwires it to
|
||||||
|
`#f`. Under this RFC the build sets it (and a parallel `whole-program?` flag)
|
||||||
|
from the chosen mode before compiling, so:
|
||||||
|
|
||||||
|
- release: `inline-enabled?` → true, whole-program off. Per-ns inference and
|
||||||
|
inlining light up; `fl*`/`fx*` stays off.
|
||||||
|
- optimized: both on; the types pass runs its whole-program fixpoint and the
|
||||||
|
back end may emit unchecked numeric ops where a flonum/fixnum is proven.
|
||||||
|
|
||||||
|
No new pass is required to reach release — it is the ported passes, ungated.
|
||||||
|
|
||||||
|
## Staging
|
||||||
|
|
||||||
|
1. **Spike (de-risk Chez AOT).** Emit a trivial whole program to disk and prove
|
||||||
|
`compile-program` + boot/static link yields a standalone binary that runs.
|
||||||
|
This is the only real unknown.
|
||||||
|
2. **`jolt build` release.** Front-end + file-emitting back-end path + flip
|
||||||
|
`inline-enabled?` from the mode. Gate against the bench/corpus suites; binary
|
||||||
|
output must pass the corpus a `run` passes.
|
||||||
|
3. **DCE.** Re-port the reachability pass; gate with a test like the old
|
||||||
|
`uberscript-dce` case.
|
||||||
|
4. **Optimized.** Whole-program flag, `compile-whole-program`, `fl*`/`fx*`
|
||||||
|
emission. Gate on the bench suite (ray tracer, binary-trees) for size and
|
||||||
|
speed vs the spike baseline.
|
||||||
|
|
||||||
|
Each stage is TDD against the existing gates (`make test`, `make corpus`, the
|
||||||
|
`bench/` programs). Modes land behind the build command, so dev — the only mode
|
||||||
|
today — is unaffected until a stage proves out.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- **Static vs. boot-file linking.** A fully static Chez image is the smallest,
|
||||||
|
most portable artifact but the most work to link; a boot file plus a stub
|
||||||
|
launcher is the easy first cut. Spike decides which step 1 targets.
|
||||||
|
- **FFI in a built binary.** `jolt.ffi` loads native libraries at runtime; a
|
||||||
|
closed-world build still needs that to work. The build must bake the FFI
|
||||||
|
Clojure side and keep dynamic `dlopen` at run time.
|
||||||
|
- **Macro and `eval` at runtime.** Release/optimized are closed-world, but an
|
||||||
|
app that calls `eval`/`load-string` needs the compiler present. Either ship
|
||||||
|
the compiler image in the binary (larger) or reject those builds (the DCE
|
||||||
|
bail-out already detects the calls).
|
||||||
|
|
||||||
|
## Prior art in this repo
|
||||||
|
|
||||||
|
The optimization design these modes turn on is RFC 0004 (type hints), RFC 0005
|
||||||
|
(structural inference), RFC 0006 (success checking). The linking model — direct
|
||||||
|
linking as a per-unit property, `^:redef`/`^:dynamic` as the only opt-out — and
|
||||||
|
the cross-ns specialization penalty are recorded in beads memories
|
||||||
|
(`jolt-linking-model`, `cross-ns-param-penalty`). Phase 4 (`jolt-cf1q.5`) is the
|
||||||
|
tracking issue.
|
||||||
32
host/chez/build-smoke.sh
Executable file
32
host/chez/build-smoke.sh
Executable file
|
|
@ -0,0 +1,32 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# build smoke: `jolt build` compiles a multi-namespace app (macro + cross-ns +
|
||||||
|
# clojure.string) into a standalone binary, which then runs with no jolt source
|
||||||
|
# or Chez install on the path — args reach -main, output matches.
|
||||||
|
root="$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd)"
|
||||||
|
cd "$root"
|
||||||
|
|
||||||
|
app="$root/test/chez/build-app"
|
||||||
|
out="$(mktemp -d)/app-bin"
|
||||||
|
trap 'rm -rf "$(dirname "$out")"' EXIT
|
||||||
|
|
||||||
|
echo "build smoke: compiling app.core -> $out"
|
||||||
|
if ! JOLT_PWD="$app" bin/joltc build -m app.core -o "$out" >/dev/null 2>&1; then
|
||||||
|
echo " FAIL: jolt build exited non-zero"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
[ -x "$out" ] || { echo " FAIL: no executable produced"; exit 1; }
|
||||||
|
|
||||||
|
# Run from a neutral cwd with args; check the three output lines.
|
||||||
|
got="$(cd / && "$out" alpha bb ccc 2>&1)"
|
||||||
|
want='HELLO FROM A BUILT BINARY!
|
||||||
|
HELLO FROM A BUILT BINARY!
|
||||||
|
args: [alpha bb ccc]
|
||||||
|
sum: 10'
|
||||||
|
if [ "$got" = "$want" ]; then
|
||||||
|
echo "build smoke: passed"
|
||||||
|
else
|
||||||
|
echo " FAIL: binary output mismatch"
|
||||||
|
echo "--- want ---"; echo "$want"
|
||||||
|
echo "--- got ----"; echo "$got"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
253
host/chez/build.ss
Normal file
253
host/chez/build.ss
Normal file
|
|
@ -0,0 +1,253 @@
|
||||||
|
;; build.ss — `jolt build`: AOT-compile an app into a standalone executable.
|
||||||
|
;;
|
||||||
|
;; Loaded on demand by cli.ss when the command is `build`. Defines the host
|
||||||
|
;; primitive jolt.host/build-binary, which jolt.main's build command calls after
|
||||||
|
;; resolving the project's deps + source roots.
|
||||||
|
;;
|
||||||
|
;; The pipeline (Phase 4 stage 2):
|
||||||
|
;; 1. load the entry namespace — registers its macros/vars and follows requires,
|
||||||
|
;; recording the app namespaces in dependency order (loader's ns-loaded-hook).
|
||||||
|
;; 2. re-emit each app namespace to Scheme (the emit-image cross-compile path),
|
||||||
|
;; now that its macros are registered.
|
||||||
|
;; 3. textually inline the cli.ss runtime load sequence into one flat source,
|
||||||
|
;; append the app emission + a launcher that calls the entry's -main.
|
||||||
|
;; 4. compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link
|
||||||
|
;; against libkernel.a into a single self-contained binary.
|
||||||
|
;;
|
||||||
|
;; emit-image.ss supplies the cross-compiler (ei-* helpers); it's loaded here so a
|
||||||
|
;; normal run never pays for it.
|
||||||
|
|
||||||
|
(load "host/chez/emit-image.ss")
|
||||||
|
|
||||||
|
;; --- shell helpers ----------------------------------------------------------
|
||||||
|
;; Run a command, return its stdout as one trimmed string ("" on no output).
|
||||||
|
(define (bld-sh-capture cmd)
|
||||||
|
(let* ((p (process cmd)) (in (car p)))
|
||||||
|
(let loop ((acc '()))
|
||||||
|
(let ((l (get-line in)))
|
||||||
|
(if (eof-object? l)
|
||||||
|
(begin (close-port in)
|
||||||
|
(let ((s (apply string-append (reverse acc))))
|
||||||
|
;; trim a trailing newline-equivalent (we joined without them)
|
||||||
|
s))
|
||||||
|
(loop (cons l acc)))))))
|
||||||
|
|
||||||
|
(define (bld-system cmd)
|
||||||
|
(let ((rc (system cmd)))
|
||||||
|
(unless (zero? rc)
|
||||||
|
(error 'jolt-build (string-append "command failed (" (number->string rc) "): " cmd)))))
|
||||||
|
|
||||||
|
(define (bld-contains? s sub)
|
||||||
|
(let ((ns (string-length s)) (nsub (string-length sub)))
|
||||||
|
(let loop ((i 0))
|
||||||
|
(cond ((> (+ i nsub) ns) #f)
|
||||||
|
((string=? (substring s i (+ i nsub)) sub) #t)
|
||||||
|
(else (loop (+ i 1)))))))
|
||||||
|
|
||||||
|
;; --- toolchain discovery ----------------------------------------------------
|
||||||
|
(define bld-machine (symbol->string (machine-type)))
|
||||||
|
(define bld-osx? (bld-contains? bld-machine "osx"))
|
||||||
|
|
||||||
|
;; The Chez executable, for the isolated compile pass (see build-binary step 4).
|
||||||
|
(define bld-chez
|
||||||
|
(let ((p (bld-sh-capture "command -v chez || command -v scheme || command -v petite")))
|
||||||
|
(if (> (string-length p) 0) p "chez")))
|
||||||
|
|
||||||
|
;; Chez version off (scheme-version) "Chez Scheme Version X.Y.Z" — last token.
|
||||||
|
(define bld-version
|
||||||
|
(let* ((s (scheme-version)) (n (string-length s)))
|
||||||
|
(let loop ((i n))
|
||||||
|
(if (or (= i 0) (char=? (string-ref s (- i 1)) #\space))
|
||||||
|
(substring s i n)
|
||||||
|
(loop (- i 1))))))
|
||||||
|
|
||||||
|
;; The csv<ver>/<machine> dir holding scheme.h, libkernel.a, *.boot. Derived from
|
||||||
|
;; the chez executable's location; JOLT_CHEZ_CSV overrides.
|
||||||
|
(define bld-csv-dir
|
||||||
|
(let ((env (getenv "JOLT_CHEZ_CSV")))
|
||||||
|
(or (and env (> (string-length env) 0) env)
|
||||||
|
(let* ((bindir (bld-sh-capture "dirname \"$(command -v chez || command -v scheme || command -v petite)\""))
|
||||||
|
(cand (string-append bindir "/../lib/csv" bld-version "/" bld-machine)))
|
||||||
|
cand))))
|
||||||
|
|
||||||
|
(define (bld-check-toolchain)
|
||||||
|
(for-each
|
||||||
|
(lambda (f)
|
||||||
|
(let ((p (string-append bld-csv-dir "/" f)))
|
||||||
|
(unless (file-exists? p)
|
||||||
|
(error 'jolt-build (string-append "Chez build file missing: " p
|
||||||
|
"\nSet JOLT_CHEZ_CSV to the csv<ver>/<machine> dir.")))))
|
||||||
|
'("scheme.h" "libkernel.a" "petite.boot" "scheme.boot")))
|
||||||
|
|
||||||
|
;; Link flags. macOS Homebrew layout for the kernel's lz4/zlib/ncurses deps.
|
||||||
|
(define (bld-link-libs)
|
||||||
|
(if bld-osx?
|
||||||
|
(let ((lz4 (bld-sh-capture "brew --prefix lz4 2>/dev/null")))
|
||||||
|
(string-append
|
||||||
|
(if (> (string-length lz4) 0) (string-append "-L" lz4 "/lib ") "")
|
||||||
|
"-llz4 -lz -lncurses -framework Foundation -liconv -lm"))
|
||||||
|
;; Best-effort Linux/other; untested here.
|
||||||
|
"-llz4 -lz -lncurses -ldl -lpthread -lm"))
|
||||||
|
|
||||||
|
;; --- runtime manifest (mirrors host/chez/cli.ss's load order) ---------------
|
||||||
|
(define bld-runtime-manifest
|
||||||
|
(list
|
||||||
|
"(load \"host/chez/rt.ss\")"
|
||||||
|
"(set-chez-ns! \"clojure.core\")"
|
||||||
|
"(load \"host/chez/seed/prelude.ss\")"
|
||||||
|
"(load \"host/chez/post-prelude.ss\")"
|
||||||
|
"(set-chez-ns! \"user\")"
|
||||||
|
"(load \"host/chez/host-contract.ss\")"
|
||||||
|
"(load \"host/chez/seed/image.ss\")"
|
||||||
|
"(load \"host/chez/compile-eval.ss\")"
|
||||||
|
"(load \"host/chez/png.ss\")"
|
||||||
|
"(load \"host/chez/loader.ss\")"
|
||||||
|
"(load \"host/chez/ffi.ss\")"
|
||||||
|
"(set-source-roots! (list \"jolt-core\" \"stdlib\"))"))
|
||||||
|
|
||||||
|
;; A single-line top-level `(load "PATH")` -> PATH, else #f.
|
||||||
|
(define (bld-load-path line)
|
||||||
|
(let ((s (let trim ((i 0))
|
||||||
|
(if (and (< i (string-length line))
|
||||||
|
(memv (string-ref line i) '(#\space #\tab)))
|
||||||
|
(trim (+ i 1))
|
||||||
|
(substring line i (string-length line))))))
|
||||||
|
(and (>= (string-length s) 7)
|
||||||
|
(string=? (substring s 0 6) "(load ")
|
||||||
|
(let* ((q1 (let scan ((i 6)) (if (char=? (string-ref s i) #\") i (scan (+ i 1)))))
|
||||||
|
(q2 (let scan ((i (+ q1 1))) (if (char=? (string-ref s i) #\") i (scan (+ i 1))))))
|
||||||
|
(substring s (+ q1 1) q2)))))
|
||||||
|
|
||||||
|
(define (bld-file-lines path)
|
||||||
|
(call-with-input-file path
|
||||||
|
(lambda (p)
|
||||||
|
(let loop ((acc '()))
|
||||||
|
(let ((l (get-line p)))
|
||||||
|
(if (eof-object? l) (reverse acc) (loop (cons l acc))))))))
|
||||||
|
|
||||||
|
;; Emit one line to OUT, recursively inlining a `(load ...)` of a repo file.
|
||||||
|
(define (bld-inline-line line out depth)
|
||||||
|
(when (> depth 50) (error 'jolt-build "load nesting too deep"))
|
||||||
|
(let ((p (bld-load-path line)))
|
||||||
|
(if p
|
||||||
|
(for-each (lambda (l) (bld-inline-line l out (+ depth 1))) (bld-file-lines p))
|
||||||
|
(begin (put-string out line) (put-string out "\n")))))
|
||||||
|
|
||||||
|
(define (bld-emit-runtime out)
|
||||||
|
(for-each (lambda (l) (bld-inline-line l out 0)) bld-runtime-manifest))
|
||||||
|
|
||||||
|
;; --- app emission -----------------------------------------------------------
|
||||||
|
;; Re-emit one app namespace to a list of Scheme strings. Like emit-image's
|
||||||
|
;; ei-emit-ns but WITHOUT the silent (guard ...) wrapper — a form that fails to
|
||||||
|
;; emit must fail the build, not vanish.
|
||||||
|
(define (bld-emit-ns ns-name src)
|
||||||
|
(let loop ((forms (ei-read-all src)) (acc '()))
|
||||||
|
(if (null? forms)
|
||||||
|
(reverse acc)
|
||||||
|
(let ((f (car forms)))
|
||||||
|
(ce-scan-requires! f ns-name)
|
||||||
|
(cond
|
||||||
|
((ei-ns-form? f) (loop (cdr forms) acc))
|
||||||
|
((ce-macro-form? f)
|
||||||
|
(let-values (((nm fn-form) (ce-defmacro->fn f)))
|
||||||
|
(let ((scm (let ((ctx (make-analyze-ctx ns-name)))
|
||||||
|
(jolt-ce-emit (jolt-ce-analyze ctx fn-form)))))
|
||||||
|
(loop (cdr forms)
|
||||||
|
(cons (string-append
|
||||||
|
"(def-var! " (ei-str-lit ns-name) " " (ei-str-lit nm) "\n "
|
||||||
|
scm ")\n(mark-macro! "
|
||||||
|
(ei-str-lit ns-name) " " (ei-str-lit nm) ")")
|
||||||
|
acc)))))
|
||||||
|
(else
|
||||||
|
(let* ((ctx (make-analyze-ctx ns-name))
|
||||||
|
(scm (jolt-ce-emit (jolt-ce-analyze ctx f))))
|
||||||
|
(loop (cdr forms) (cons scm acc)))))))))
|
||||||
|
|
||||||
|
;; --- the build --------------------------------------------------------------
|
||||||
|
;; entry-ns: the app's main namespace (a string). out-path: the binary to write.
|
||||||
|
;; mode: "dev" | "release" | "optimized" (recorded; optimization passes wired in a
|
||||||
|
;; later stage). Deps + source roots are already applied by the caller.
|
||||||
|
(define (build-binary entry-ns out-path mode)
|
||||||
|
(bld-check-toolchain)
|
||||||
|
;; 1. record app namespaces in dependency order as they finish loading.
|
||||||
|
(let ((app-order '()))
|
||||||
|
(set-ns-loaded-hook!
|
||||||
|
(lambda (name file) (set! app-order (cons (cons name file) app-order))))
|
||||||
|
(load-namespace entry-ns)
|
||||||
|
(set-ns-loaded-hook! (lambda (name file) #f))
|
||||||
|
(let ((ordered (reverse app-order))) ; deps first, entry last
|
||||||
|
(when (null? ordered)
|
||||||
|
(error 'jolt-build (string-append "no source namespace loaded for " entry-ns
|
||||||
|
" — is it on the source roots?")))
|
||||||
|
;; 2. emit each app namespace.
|
||||||
|
(let* ((app-strs (apply append
|
||||||
|
(map (lambda (nf) (bld-emit-ns (car nf) (read-file-string (cdr nf))))
|
||||||
|
ordered)))
|
||||||
|
(builddir (string-append out-path ".build"))
|
||||||
|
(flat-ss (string-append builddir "/flat.ss"))
|
||||||
|
(flat-so (string-append builddir "/flat.so"))
|
||||||
|
(boot (string-append builddir "/jolt.boot"))
|
||||||
|
(boot-h (string-append builddir "/boot_data.h"))
|
||||||
|
(main-c (string-append builddir "/main.c")))
|
||||||
|
(bld-system (string-append "mkdir -p '" builddir "'"))
|
||||||
|
;; 3. flat source = runtime + app + launcher.
|
||||||
|
(let ((out (open-output-file flat-ss 'replace)))
|
||||||
|
(bld-emit-runtime out)
|
||||||
|
(put-string out "\n;; === app ===\n")
|
||||||
|
(for-each (lambda (s) (put-string out s) (put-string out "\n")) app-strs)
|
||||||
|
;; The launcher runs as Chez's scheme-start (so argv reaches -main —
|
||||||
|
;; top-level boot forms run during heap build, before args are set), and
|
||||||
|
;; suppresses the interactive greeting.
|
||||||
|
(put-string out "\n;; === launcher ===\n")
|
||||||
|
(put-string out (string-append
|
||||||
|
"(suppress-greeting #t)\n"
|
||||||
|
"(scheme-start\n"
|
||||||
|
" (lambda args\n"
|
||||||
|
" (let ((mainv (var-deref " (ei-str-lit entry-ns) " \"-main\")))\n"
|
||||||
|
" (apply jolt-invoke mainv args))\n"
|
||||||
|
" (exit 0)))\n"))
|
||||||
|
(close-port out))
|
||||||
|
;; 4. compile -> boot -> embed -> link.
|
||||||
|
;; compile-file/make-boot-file run in a FRESH Chez, not this process: the
|
||||||
|
;; loaded runtime shadows `error` (regex.ss, for irregex), which would
|
||||||
|
;; otherwise bake a broken `error` reference into the boot.
|
||||||
|
(display (string-append "jolt build: compiling " entry-ns " (" mode " mode)\n"))
|
||||||
|
(let ((cs (string-append builddir "/compile.ss")))
|
||||||
|
(let ((p (open-output-file cs 'replace)))
|
||||||
|
(put-string p
|
||||||
|
(string-append
|
||||||
|
"(import (chezscheme))\n"
|
||||||
|
"(compile-file " (ei-str-lit flat-ss) " " (ei-str-lit flat-so) ")\n"
|
||||||
|
"(make-boot-file " (ei-str-lit boot) " '()\n "
|
||||||
|
(ei-str-lit (string-append bld-csv-dir "/petite.boot")) "\n "
|
||||||
|
(ei-str-lit (string-append bld-csv-dir "/scheme.boot")) "\n "
|
||||||
|
(ei-str-lit flat-so) ")\n"))
|
||||||
|
(close-port p))
|
||||||
|
(bld-system (string-append bld-chez " --script '" cs "'")))
|
||||||
|
(bld-system (string-append "xxd -i '" boot "' > '" boot-h "'"))
|
||||||
|
;; The xxd symbol is derived from the path; normalize to jolt_boot.
|
||||||
|
(bld-system (string-append
|
||||||
|
"sed -i.bak -E 's/unsigned char [A-Za-z0-9_]+\\[\\]/unsigned char jolt_boot[]/; "
|
||||||
|
"s/unsigned int [A-Za-z0-9_]+_len/unsigned int jolt_boot_len/' '" boot-h "'"))
|
||||||
|
(let ((mc (open-output-file main-c 'replace)))
|
||||||
|
(put-string mc
|
||||||
|
(string-append
|
||||||
|
"#include \"scheme.h\"\n#include \"boot_data.h\"\n"
|
||||||
|
"int main(int argc, char *argv[]) {\n"
|
||||||
|
" Sscheme_init(0);\n"
|
||||||
|
" Sregister_boot_file_bytes(\"jolt\", jolt_boot, jolt_boot_len);\n"
|
||||||
|
" Sbuild_heap(0, 0);\n"
|
||||||
|
" int status = Sscheme_start(argc, (const char **)argv);\n"
|
||||||
|
" Sscheme_deinit();\n return status;\n}\n"))
|
||||||
|
(close-port mc))
|
||||||
|
(bld-system (string-append
|
||||||
|
"cc -O2 -I'" bld-csv-dir "' '" main-c "' '" bld-csv-dir "/libkernel.a' "
|
||||||
|
"-o '" out-path "' " (bld-link-libs)))
|
||||||
|
(display (string-append "jolt build: wrote " out-path "\n"))))))
|
||||||
|
|
||||||
|
(def-var! "jolt.host" "build-binary"
|
||||||
|
(lambda (entry out mode)
|
||||||
|
(build-binary (jolt-str-render-one entry)
|
||||||
|
(jolt-str-render-one out)
|
||||||
|
(jolt-str-render-one mode))
|
||||||
|
jolt-nil))
|
||||||
|
|
@ -66,6 +66,11 @@
|
||||||
(display result) (newline))))
|
(display result) (newline))))
|
||||||
;; otherwise dispatch the argv through jolt.main/-main
|
;; otherwise dispatch the argv through jolt.main/-main
|
||||||
(else
|
(else
|
||||||
|
;; `build` AOT-compiles an app to a standalone binary — load the build
|
||||||
|
;; driver (the cross-compiler emitter) on demand so a normal run never pays
|
||||||
|
;; for it. It defines jolt.host/build-binary, which jolt.main's build cmd calls.
|
||||||
|
(when (and (pair? cli-args) (string=? (car cli-args) "build"))
|
||||||
|
(load "host/chez/build.ss"))
|
||||||
(load-namespace "jolt.main")
|
(load-namespace "jolt.main")
|
||||||
(let ((mainv (var-deref "jolt.main" "-main")))
|
(let ((mainv (var-deref "jolt.main" "-main")))
|
||||||
(apply jolt-invoke mainv cli-args)))))
|
(apply jolt-invoke mainv cli-args)))))
|
||||||
|
|
|
||||||
|
|
@ -53,6 +53,23 @@
|
||||||
(vector-for-each (lambda (c) (hashtable-set! loaded-ns (var-cell-ns c) #t))
|
(vector-for-each (lambda (c) (hashtable-set! loaded-ns (var-cell-ns c) #t))
|
||||||
(hashtable-values var-table))
|
(hashtable-values var-table))
|
||||||
|
|
||||||
|
;; Does `name` already have vars in the var-table? A namespace baked into the
|
||||||
|
;; image after the snapshot above — an AOT'd app namespace in a `jolt build`
|
||||||
|
;; binary — exists in memory with no source file; a later `require` of it must
|
||||||
|
;; no-op rather than hunt the (absent) source.
|
||||||
|
(define (ns-has-vars? name)
|
||||||
|
(let ((found #f))
|
||||||
|
(vector-for-each
|
||||||
|
(lambda (c) (when (and (not found) (string=? (var-cell-ns c) name)) (set! found #t)))
|
||||||
|
(hashtable-values var-table))
|
||||||
|
found))
|
||||||
|
|
||||||
|
;; Called after a file-backed namespace finishes loading, with (name file). The
|
||||||
|
;; build driver sets this to record app namespaces in dependency order for AOT
|
||||||
|
;; emission; a no-op for normal runs.
|
||||||
|
(define ns-loaded-hook (lambda (name file) #f))
|
||||||
|
(define (set-ns-loaded-hook! f) (set! ns-loaded-hook f))
|
||||||
|
|
||||||
;; Read every form from a file and compile+eval it in turn. The first form is
|
;; Read every form from a file and compile+eval it in turn. The first form is
|
||||||
;; normally (ns …), which expands to (in-ns …) and switches the current ns, so
|
;; normally (ns …), which expands to (in-ns …) and switches the current ns, so
|
||||||
;; later forms compile in that namespace — (chez-current-ns) is re-read each step.
|
;; later forms compile in that namespace — (chez-current-ns) is re-read each step.
|
||||||
|
|
@ -80,17 +97,22 @@
|
||||||
;; restored afterward, since loading the file switched it.
|
;; restored afterward, since loading the file switched it.
|
||||||
(define (load-namespace name)
|
(define (load-namespace name)
|
||||||
(unless (hashtable-ref loaded-ns name #f)
|
(unless (hashtable-ref loaded-ns name #f)
|
||||||
(hashtable-set! loaded-ns name #t)
|
|
||||||
(let ((file (find-ns-file name)))
|
(let ((file (find-ns-file name)))
|
||||||
(if (not file)
|
(cond
|
||||||
(begin
|
(file
|
||||||
(hashtable-delete! loaded-ns name)
|
(hashtable-set! loaded-ns name #t) ; mark before load so a cycle terminates
|
||||||
(error #f (string-append "Could not locate " (ns-name->rel name)
|
(let ((saved (chez-current-ns)))
|
||||||
".clj (or .cljc) on the source roots") name))
|
(load-jolt-file file)
|
||||||
(let ((saved (chez-current-ns)))
|
;; restore the current ns (thread-local); *ns* reads derive from it.
|
||||||
(load-jolt-file file)
|
(set-chez-ns! saved))
|
||||||
;; restore the current ns (thread-local); *ns* reads derive from it.
|
(ns-loaded-hook name file))
|
||||||
(set-chez-ns! saved))))))
|
;; No source file but the namespace exists in memory (AOT'd into a built
|
||||||
|
;; binary): it's already defined — mark loaded and move on.
|
||||||
|
((ns-has-vars? name)
|
||||||
|
(hashtable-set! loaded-ns name #t))
|
||||||
|
(else
|
||||||
|
(error #f (string-append "Could not locate " (ns-name->rel name)
|
||||||
|
".clj (or .cljc) on the source roots") name))))))
|
||||||
|
|
||||||
;; load-file: load an explicit path (a `run FILE`), in the current ns.
|
;; load-file: load an explicit path (a `run FILE`), in the current ns.
|
||||||
(define (jolt-load-file path)
|
(define (jolt-load-file path)
|
||||||
|
|
|
||||||
|
|
@ -112,6 +112,28 @@
|
||||||
(map? task) (do (apply-project! resolved) (apply-main-opts (:main-opts task) more))
|
(map? task) (do (apply-project! resolved) (apply-main-opts (:main-opts task) more))
|
||||||
:else (throw (ex-info (str "bad task " name) {})))))
|
:else (throw (ex-info (str "bad task " name) {})))))
|
||||||
|
|
||||||
|
;; build [-m NS | FILE] [-o OUT] [--opt | --dev] — AOT-compile the app into a
|
||||||
|
;; standalone executable. Resolves deps + roots like `run`, then hands the entry
|
||||||
|
;; namespace to the host build driver (jolt.host/build-binary, defined by
|
||||||
|
;; build.ss). Default mode is release; --opt selects optimized, --dev unoptimized.
|
||||||
|
(defn- cmd-build [more]
|
||||||
|
(apply-project! (deps/resolve-project (project-dir)))
|
||||||
|
(let [opts (loop [a more, entry nil, out nil]
|
||||||
|
(cond
|
||||||
|
(empty? a) {:entry entry :out out}
|
||||||
|
(= "-m" (first a)) (recur (drop 2 a) (second a) out)
|
||||||
|
(= "-o" (first a)) (recur (drop 2 a) entry (second a))
|
||||||
|
(str/starts-with? (first a) "-") (recur (rest a) entry out)
|
||||||
|
:else (recur (rest a) (or entry (first a)) out)))
|
||||||
|
entry (:entry opts)
|
||||||
|
mode (cond (some #{"--opt"} more) "optimized"
|
||||||
|
(some #{"--dev"} more) "dev"
|
||||||
|
:else "release")]
|
||||||
|
(when (nil? entry)
|
||||||
|
(throw (ex-info "build needs an entry: -m NS" {})))
|
||||||
|
(let [out (or (:out opts) (first (str/split entry #"\.")))]
|
||||||
|
(jolt.host/build-binary entry out mode))))
|
||||||
|
|
||||||
(defn- nrepl [more]
|
(defn- nrepl [more]
|
||||||
;; resolve the project (deps on the roots, native libs loaded), then start the
|
;; resolve the project (deps on the roots, native libs loaded), then start the
|
||||||
;; nREPL server so an editor can connect and (require '[some.lib]) live. A
|
;; nREPL server so an editor can connect and (require '[some.lib]) live. A
|
||||||
|
|
@ -128,6 +150,7 @@
|
||||||
(println "usage: jolt <command> [args]")
|
(println "usage: jolt <command> [args]")
|
||||||
(println " run -m NS [args] resolve deps.edn, load NS, call its -main")
|
(println " run -m NS [args] resolve deps.edn, load NS, call its -main")
|
||||||
(println " run FILE load a Clojure file")
|
(println " run FILE load a Clojure file")
|
||||||
|
(println " build -m NS [-o OUT] [--opt|--dev] compile a standalone binary")
|
||||||
(println " -M:alias [args] run the alias's :main-opts")
|
(println " -M:alias [args] run the alias's :main-opts")
|
||||||
(println " -A:alias [args] add the alias's paths/deps")
|
(println " -A:alias [args] add the alias's paths/deps")
|
||||||
(println " repl start a line REPL")
|
(println " repl start a line REPL")
|
||||||
|
|
@ -146,4 +169,5 @@
|
||||||
(str/starts-with? cmd "-M") (cmd-M cmd more)
|
(str/starts-with? cmd "-M") (cmd-M cmd more)
|
||||||
(str/starts-with? cmd "-A") (cmd-A cmd more)
|
(str/starts-with? cmd "-A") (cmd-A cmd more)
|
||||||
(= cmd "-m") (cmd-run (cons "-m" more))
|
(= cmd "-m") (cmd-run (cons "-m" more))
|
||||||
|
(= cmd "build") (cmd-build more)
|
||||||
:else (run-task cmd more))))
|
:else (run-task cmd more))))
|
||||||
|
|
|
||||||
1
test/chez/build-app/deps.edn
Normal file
1
test/chez/build-app/deps.edn
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
{:paths ["src"]}
|
||||||
7
test/chez/build-app/src/app/core.clj
Normal file
7
test/chez/build-app/src/app/core.clj
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
(ns app.core
|
||||||
|
(:require [app.util :as util]))
|
||||||
|
|
||||||
|
(defn -main [& args]
|
||||||
|
(util/twice (println (util/shout "hello from a built binary")))
|
||||||
|
(println "args:" (vec args))
|
||||||
|
(println "sum:" (reduce + (map count args))))
|
||||||
8
test/chez/build-app/src/app/util.clj
Normal file
8
test/chez/build-app/src/app/util.clj
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
(ns app.util
|
||||||
|
(:require [clojure.string :as str]))
|
||||||
|
|
||||||
|
(defn shout [s]
|
||||||
|
(str/upper-case (str s "!")))
|
||||||
|
|
||||||
|
(defmacro twice [x]
|
||||||
|
`(do ~x ~x))
|
||||||
Loading…
Add table
Add a link
Reference in a new issue