Docs: Chez-only, drop the Janet-era references and obsolete migration notes
Bring the docs in line with the actual implementation now that Chez is the sole substrate. Deleted the migration/spike/handoff artifacts that only documented the Janet era or the port effort: the port plan, phase-0 and foundational-runtime spike writeups (+ the stray root-level copy), the self-hosting design notes, the architecture-refactor plan, and spike/chez/RESULTS.md. Rewrote the current reference docs against the Chez facts: building-and-deps and tools-deps (no jpm/build step — bin/joltc off the checked-in seed, deps via jolt.deps into ~/.jolt/gitlibs), libraries (SQLite is built-in jdbc.core over libsqlite3, not a Janet driver), the conformance/spec test-flow docs (the Chez corpus runner + certify, no .janet harnesses), and the transient / type-hint / seed-overlay design notes (Chez representations: mutable transients, flat copy-on-write vectors, HAMT maps, the seed/overlay twin). Fixed the README collections line (vectors aren't 32-way tries) and added the ffi/transient gate targets. rfc 0001's numerics open-question is resolved (the Scheme tower). Renamed the built-in HTTP adapter to jolt.http.server only (dropped the ring-janet.adapter alias — a Janet-era name).
This commit is contained in:
Yogthos
parent
fe3fdf6b9c
commit
45876998ad
28 changed files with 253 additions and 2012 deletions
|
|
@ -1,267 +0,0 @@
|
|||
# Architecture Refactor Plan
|
||||
|
||||
Goal: make the jolt codebase easier to understand and safe to change — for human
|
||||
maintainers and, specifically, for LLM agents. An agent should be able to find
|
||||
where a feature lives, see its related code, and make a change without scanning
|
||||
3000-line files or keeping invisible global state in mind.
|
||||
|
||||
This plan synthesizes a six-part architectural review (one reviewer per
|
||||
subsystem). It is organized as **independent, gate-validated phases**, ordered by
|
||||
value-to-risk. Each phase is a PR-sized unit. Nothing here changes behavior; it is
|
||||
pure reorganization plus a small set of dead-code/bug deletions.
|
||||
|
||||
## Non-negotiable constraints (apply to every phase)
|
||||
|
||||
- **Every phase passes the full gate**: `jpm test` green (conformance ×3 modes,
|
||||
`clojure-test-suite` ≥ baseline, bench back-to-back vs main, real exit code).
|
||||
`rm -rf build && jpm clean` before trusting the binary.
|
||||
- **Load order is load-bearing.** The Janet seed (`src/jolt/*.janet`) and the
|
||||
Clojure overlay tiers (`jolt-core/clojure/core/NN-*.clj`) load in a fixed order;
|
||||
a module may only use what loads before it. Splitting a file must preserve the
|
||||
net load order (a new file is imported where the old code ran).
|
||||
- **Seed-tier discipline** (the jolt-tzo rules in CLAUDE.md): nothing the
|
||||
analyzer/IR use may move below the kernel tier; a tier may only use macros from
|
||||
earlier tiers; expander-called fns stay in 00-syntax.
|
||||
- **One concern per PR.** Do not combine a file split with a behavior fix.
|
||||
Dead-code/bug deletions (Phase 0) land first and separately so later diffs are
|
||||
pure moves.
|
||||
|
||||
## Guiding principles (the target state)
|
||||
|
||||
1. **One obvious home per feature.** Adding a `.someMethod` interop shim, a reader
|
||||
macro, or a clojure.core fn should have exactly one file an agent edits.
|
||||
2. **Files map to concerns, not to history.** No 3000-line grab-bags; no module
|
||||
whose name lies about its contents (`javatime.janet` is 80% not java.time;
|
||||
`phm.janet` contains LazySeq; `types.janet` holds seven concerns).
|
||||
3. **Make implicit contracts explicit and checked.** The seed↔overlay split, the
|
||||
ctx-shaping env-knob list, and the IR op set are tribal knowledge today; turn
|
||||
each into a single source of truth with a drift check.
|
||||
4. **No copy-paste dispatch.** Where the same op-set / member-dispatch / cache
|
||||
dance is hand-written N times, extract one combinator.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Dead code & concrete bugs (low risk, do first)
|
||||
|
||||
Pure deletions and small fixes, each with a regression row. These remove
|
||||
*actively misleading* code (comments that contradict behavior) before any moves.
|
||||
|
||||
| Item | Location | Note |
|
||||
|---|---|---|
|
||||
| `find` defined twice; the dead copy returns a plain vector and is *wrong* | `jolt-core/clojure/core/20-coll.clj:347-349` | live def at :787 returns a real map-entry; delete the dead one |
|
||||
| `core-satisfies?` always returns `false` (latent bug + misleading comment) | `src/jolt/core.janet:2412` | either implement over the protocol registry or document why inert; fix the `eval-list` comment that claims it's an overlay fn |
|
||||
| `File.toURL` stores `:url` but every `:jolt/url` method reads `:spec` | `src/jolt/javatime.janet:636` | broken shim; use `:spec`, add a spec row |
|
||||
| `core-type->str` — zero references | `src/jolt/core.janet:2416` | delete defn + binding |
|
||||
| `core-resolve` — unreachable (overridden by `install-stateful-fns!`) | `src/jolt/core.janet:2365` | delete defn + binding; fix stale comment |
|
||||
| `mark-hint` — unreachable | `jolt-core/jolt/passes.clj:836` | delete |
|
||||
| `pad2` defined twice | `src/jolt/javatime.janet:41,522` | keep one |
|
||||
| `phs-to-struct`, `shape-vals`, `ns-imports-fn` — zero references | `phm.janet:302`, `types.janet:655`, `types.janet:527` | delete unless reserved API |
|
||||
| `pl-rest` no-op `(if (plist? r) r r)` | `src/jolt/plist.janet:62` | collapse / fix; regression check |
|
||||
| `read-quote` unused `pos` param | `src/jolt/reader.janet:608` | drop |
|
||||
| Stale/contradictory comments (`extend`, orphan `;; trampoline:` / `;; rand-int:` headers, migration breadcrumbs) | `30-macros.clj:402-404`, `20-coll.clj:558-560,780,790` | sweep |
|
||||
| **`:map-shapes?` missing from the deps-image cache key** (possible stale-image correctness gap) | `src/jolt/main.janet:440-448` | add to key; confirm with a test |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Extract the host-interop subsystem (highest value)
|
||||
|
||||
**Problem (corroborated by 3 reviewers).** The JVM-emulation shim layer is the
|
||||
single worst sprawl, and it is exactly where the recent hiccup/markdown/malli work
|
||||
landed ad hoc. A single class is split across up to three files:
|
||||
|
||||
- registry *machinery* (`class-statics`/`tagged-methods`/`class-ctors` +
|
||||
`register-*!`) lives in `evaluator.janet:624-651`;
|
||||
- four static tables (`Math`/`Thread`/`System`/`Long`/`Number`) are hardcoded in
|
||||
`evaluator.janet:537-619` and dispatched *outside* the registry in `resolve-sym`
|
||||
(780-794); instance-method tables (`string/number/object-methods`, 698-775) are
|
||||
likewise inline;
|
||||
- the bulk of the shims are in `javatime.janet` (674 lines, ~80% not java.time:
|
||||
java.io/util/net/nio/sql/text/math all in one `install-io!`);
|
||||
- `api.janet:18-56` wires the malli statics + `set-coll-interop!` as load-time
|
||||
side effects in the public-API module;
|
||||
- `core.janet` holds `File`/JDBC constructors; `types.janet` holds the `:jolt/inst`
|
||||
representation.
|
||||
|
||||
**Target.** A `src/jolt/interop/` directory, one mechanism file + one file per JDK
|
||||
area, each owning a class's *full* surface (ctor + statics + methods + `instance?`
|
||||
predicate + canonical name) and exposing one `install!`:
|
||||
|
||||
```
|
||||
src/jolt/interop/
|
||||
registry.janet # MOVED from evaluator: class-statics/tagged-methods/class-ctors,
|
||||
# register-*!, canonical-names, value-overrides, instance? registry
|
||||
coerce.janet # shared: chr, char->byte, render-piece/writer-piece, pad2,
|
||||
# the date-format token walker (kills the pad2/render-piece dups)
|
||||
java_time.janet java_io.janet java_net.janet java_util.janet
|
||||
java_lang.janet # absorbs evaluator's inline Math/Thread/System/Long/Number +
|
||||
# string/number/object method tables — the registry becomes the
|
||||
# ONLY static/method-dispatch mechanism
|
||||
jdbc.janet
|
||||
install.janet # (install-all! ctx) — the ONE place api.janet calls (replaces api 18-56)
|
||||
```
|
||||
|
||||
`evaluator.janet`'s `resolve-sym` and the dot-dispatch consult only the registry.
|
||||
`regex.janet` and `async.janet` stay put — they are engines/library-ports consumed
|
||||
by interop, not shims.
|
||||
|
||||
**Cheap 80% if the full split is too big for one PR:** rename
|
||||
`javatime.janet`→`host_interop.janet`, pull the four inline static tables + three
|
||||
method tables out of `evaluator.janet` into it, move `api.janet:18-56` into it,
|
||||
dedup `pad2`/`chr`/`render-piece`. Collapses the scatter from 5 files to 2.
|
||||
|
||||
**Risk:** medium — touches the hot dot-dispatch path; load order must keep the
|
||||
registry available before any `install!`. Validate with `host-interop-spec` +
|
||||
the hiccup/markdown/malli example apps + full gate.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Decompose the god-files
|
||||
|
||||
The three biggest interpreter/runtime files are the top LLM-navigability tax.
|
||||
Split along the cohesive clusters the review mapped (line ranges in the review
|
||||
notes). Each split is a mechanical move + import; behavior unchanged.
|
||||
|
||||
### 2a. `evaluator.janet` (2681 → ~5 files)
|
||||
- `special_forms.janet` — explode the **680-line `eval-list`** (1921-2599) into
|
||||
named `eval-<form>` fns (`eval-fn*`, `eval-let*`, `eval-try`, `eval-dot`, …) +
|
||||
a dispatch table. This is the highest-leverage single change: today "where is
|
||||
`try` handled" means scanning a 680-line body.
|
||||
- `resolution.janet` — `resolve-sym`/`resolve-var`/binding/destructuring.
|
||||
- `ns_loader.janet` — module/bridge plumbing + require/in-ns/use/import/refer.
|
||||
- `runtime_registration.janet` — protocol/defmulti/deftype/reify setup +
|
||||
`install-stateful-fns!` (1506-1909) split into per-domain registration fns.
|
||||
- (host-interop already left in Phase 1.)
|
||||
- **De-dup the two `.method` dot arms** (2456-2507 vs 2512-2550): extract one
|
||||
`dispatch-member [target field-name args]` used by both. They are copy-pasted
|
||||
today and must be hand-synced on every interop change.
|
||||
|
||||
### 2b. `core.janet` (3017 → ~6 leaf files)
|
||||
Clusters are already sequential and mostly leaf — low risk:
|
||||
`core-types` (predicates/eq/arith/bits), `core-coll` (assoc/seq/transducers/lazy/
|
||||
transients), `core-print` (pr-str/str), `core-io`, `core-refs` (atoms/vars/delays/
|
||||
arrays/type), `core-bindings` (the table + `init-core!` + a *labeled* stub
|
||||
section). The `core-bindings` table stays the single registration point.
|
||||
|
||||
### 2c. `passes.clj` (1487 → 3-4 files behind one façade)
|
||||
The Louvain communities and the cycle analysis agree: the IR-rewriting passes and
|
||||
the type subsystem are weakly coupled (only `run-passes` + `dirty` shared).
|
||||
- `jolt/passes.clj` (keep, ~150 lines): `run-passes` + shared state + re-exports —
|
||||
the only file the back end imports.
|
||||
- `jolt/passes/fold.clj` — const-fold (always-on).
|
||||
- `jolt/passes/inline.clj` — inline + flatten-lets + scalar-replace (share the
|
||||
alpha-rename invariant).
|
||||
- `jolt/passes/types.clj` — type lattice + `infer` + success-checker + driver API
|
||||
(kept as one module to respect the inference↔checker cycle; no `declare`
|
||||
gymnastics). Extract the `infer` `:invoke` arm (1051-1160) into per-shape
|
||||
helpers regardless of the split.
|
||||
- Fix the **stale ns docstring** (lists 4 passes, omits the type system that is
|
||||
>50% of the file).
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Kill the structural duplication
|
||||
|
||||
### 3a. One IR op-walk combinator
|
||||
There are **11 hand-written recursive walks** over the IR op set (const-fold,
|
||||
inline, subst, body-closed?, pure?, flatten-lets, local-escapes?, subst-lookup,
|
||||
scalar-replace, infer, backend `emit`). Adding an IR op means editing all of them,
|
||||
and the "unknown ops pass through" promise is only partly kept. Introduce one
|
||||
`map-ir-children` (in `ir.clj` or `jolt/ir/walk.clj`) that knows each op's child
|
||||
positions; rewrite the walks as `(map-ir-children f node)` + their few specials.
|
||||
Collapses ~400 lines and makes adding an op a one-site change.
|
||||
|
||||
### 3b. IR shape hygiene
|
||||
`:let`/`:loop` bindings are `[name init]` vectors, `:map` pairs are `[k v]`, `:fn`
|
||||
arities are maps; optional keys are present-or-absent, which is *why* the backend
|
||||
needs `norm-node`/phm-densification everywhere. Make `ir.clj` constructors always
|
||||
emit optional keys (nil-valued); delete the defensive `norm-node` calls.
|
||||
|
||||
### 3c. Smaller dedups
|
||||
- `read-delimited` driver for the 4 collection readers (`read-list/vec/set/kvs`)
|
||||
in `reader.janet` (the skip/splice logic already drifted once).
|
||||
- `bucket-index-of` for the 5 stride-2 bucket scans in `phm.janet`.
|
||||
- Unify the kw-lookup head-matching reimplemented in analyzer/passes/backend
|
||||
(a kw-lookup the inference tags but the backend doesn't specialize is a silent
|
||||
miss) behind one shared predicate + fn-name table.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Config & caching coherence
|
||||
|
||||
### 4a. Lift run-mode/config resolution into `config.janet`
|
||||
`config.janet` is 15 lines (one constant); meanwhile `main.janet:585-630` holds
|
||||
~45 lines of pure env-knob policy (`open-mode?`/`dl`/`optimize?`/shape/whole-program
|
||||
gates) that can't be unit-tested without the CLI and that the cache keys need to
|
||||
share. Promote it: `config/resolve-run-mode [argv env] → {:direct-linking? :inline?
|
||||
:shapes? :map-shapes? :whole-program? :direct-link-auto?}`, plus a canonical
|
||||
`ctx-shaping-env-vars` list. `main` shrinks to: parse argv → resolve → install →
|
||||
dispatch.
|
||||
|
||||
### 4b. One ctx-image module, two policies
|
||||
`init-cached` (core image, api.janet) and the deps-image (main.janet) duplicate the
|
||||
fork→validate→reinstall-print-cb→atomic-publish dance, each with a **hand-built
|
||||
positional `%q|...` cache key** that silently misaligns if a knob is added.
|
||||
Extract `ctx_image.janet`: `load-image [path predicate]`, `save-image [ctx path]`,
|
||||
`ctx-cache-key [pairs]` (derived from `ctx-shaping-env-vars`, impossible to
|
||||
misalign). Both callers differ only in the validity predicate (source-fingerprint
|
||||
vs mtime-manifest). This also closes the cache-key footgun for good and resolves
|
||||
`aot.janet`'s status (fold its marshal helpers in, or delete it — it is exercised
|
||||
only by one integration test and is on no run path).
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Data structures, reader, types, and the seed↔overlay index
|
||||
|
||||
### 5a. `types.janet` (699 → 3-4 files)
|
||||
Seven concerns share a generic name. Split: `value.janet` (char/inst/uuid),
|
||||
keep the cohesive `var`+`ns`+`ctx` core, `protocols.janet` (the protocol/type
|
||||
registry, `type-satisfies?`), `records.janet` (shape-records). Minimum win:
|
||||
extract the protocol registry + shape-records (the two least "types"-like).
|
||||
|
||||
### 5b. `phm.janet` (303 → 3 files)
|
||||
Split out `lazyseq.janet` (LazySeq has nothing to do with hash maps — pure
|
||||
mislabel) and `phs.janet` (PersistentHashSet). `phm.janet` keeps the map.
|
||||
|
||||
### 5c. Internal collection protocol
|
||||
phm/pv/plist each re-implement count/seq/conj/predicate/meta with diverging
|
||||
naming, and `core.janet` dispatches on them with giant per-op `cond`s (every new
|
||||
structure edits every cond). A minimal `:jolt/type`-keyed vtable (`-count -seq
|
||||
-conj`) lets core dispatch once. Normalize the trio's naming (`pv` vs `pvec`,
|
||||
`->`/`-to-`, `EMPTY` vs `EMPTY-PLIST`).
|
||||
|
||||
### 5d. Make the seed↔overlay boundary self-documenting
|
||||
Five fns exist in both the seed (`core-X`) and overlay (`X`) as dispatch twins with
|
||||
no cross-reference; nothing indexes which copy is authoritative (`transduce` is
|
||||
overlay-public but `into` is seed-public — surprising and undocumented). Add:
|
||||
- a generated `REGISTRY` (name → home → public-source → seed-twin? → dispatch-only?)
|
||||
with a build-time drift check;
|
||||
- `SEED-TWIN:` provenance comments on each twin (greppable);
|
||||
- a distinct prefix for dispatch-only seed helpers so they don't read like public
|
||||
ones. Mirrors the existing "delete the seed defn + binding in the same change"
|
||||
rule.
|
||||
|
||||
### 5e. Boundary doc-comments
|
||||
Add rep-vs-API pointers between the data structures and `core.janet` (e.g. "the
|
||||
persistent vector trie lives in `pv.janet`; Clojure-facing vector ops and
|
||||
tuple/pvec polymorphism live in `core.janet`"), so an agent grepping "vector" in
|
||||
core knows where the representation is.
|
||||
|
||||
---
|
||||
|
||||
## Sequencing & rationale
|
||||
|
||||
```
|
||||
Phase 0 (dead code/bugs) — independent, do first, unblocks clean diffs
|
||||
Phase 1 (host-interop extract) — highest value; isolates the recent shim sprawl
|
||||
Phase 2 (god-file splits) — biggest navigability win; 2a/2b/2c independent
|
||||
Phase 3 (op-walk + IR hygiene) — removes the largest single duplication tax
|
||||
Phase 4 (config + caching) — fixes the cache-key footgun; makes boot legible
|
||||
Phase 5 (data/reader/types + index) — finishes the "one home per feature" goal
|
||||
```
|
||||
|
||||
Phases are independent; within a phase the sub-items (2a/2b/2c, etc.) are separate
|
||||
PRs. Highest LLM-friendliness per unit risk: **Phase 1**, **2a (`eval-list` split)**,
|
||||
**3a (op-walk combinator)**, and **5d (seed↔overlay index)**.
|
||||
|
||||
Each PR: one concern, full gate green, no behavior change (Phase 0 deletions carry
|
||||
a regression row).
|
||||
|
|
@ -1,40 +1,32 @@
|
|||
# Building and dependencies
|
||||
|
||||
How to build Jolt from source and how to pull Clojure libraries into a project.
|
||||
How to run Jolt from source and how to pull Clojure libraries into a project.
|
||||
|
||||
## Building
|
||||
## Running
|
||||
|
||||
```bash
|
||||
git clone https://github.com/jolt-lang/jolt.git
|
||||
cd jolt
|
||||
git submodule update --init # vendor/sci (used by the SCI bootstrap tests)
|
||||
jpm build
|
||||
bin/joltc -e '(println "hello")'
|
||||
```
|
||||
|
||||
This produces `build/jolt` — one binary that is both the runtime (REPL,
|
||||
file/expr runner, nREPL server) and the dependency front-end (`deps.edn`
|
||||
resolution, see below). The whole `.clj` standard library
|
||||
(`clojure.string`/`set`/`walk`/`edn`/`zip`, `jolt.http`/`interop`/`shell`/
|
||||
`nrepl`) is baked in at build time, so it loads from any directory — the artifact
|
||||
is self-contained. (`clojure.core` is built into the runtime in Janet and
|
||||
auto-referred, so it's always available.)
|
||||
There is **no build step**. `bin/joltc` (`host/chez/cli.ss`) loads the
|
||||
checked-in bootstrap seed (`host/chez/seed/{prelude,image}.ss`) plus the spine
|
||||
and compiles+evals on Chez (read → analyze → IR → emit → eval), so a fresh
|
||||
clone runs immediately. The whole `.clj` standard library
|
||||
(`clojure.string`/`set`/`walk`/`edn`/`pprint`/…) and `clojure.core` are part of
|
||||
the overlay, so they're always available.
|
||||
|
||||
The runtime **core** stays deps-agnostic: it only reads source roots from
|
||||
`JOLT_PATH`. Dependency resolution lives in a separate CLI front-end module
|
||||
(`src/jolt/deps.janet`) that the `jolt` entry point calls *before* running your
|
||||
code, and that lazily loads `jpm` (for git fetch + cache) only when it actually
|
||||
resolves. So a run with no `deps.edn` never touches the resolver, and an app
|
||||
baked from its own entry — which imports `jolt/api`, not the CLI — never links
|
||||
it at all. (`build/` also contains a `jolt-deps` shim that just forwards to
|
||||
`jolt` so old scripts keep working; prefer calling `jolt` directly.)
|
||||
`bin/joltc` is both the runtime (REPL, file/expr runner) and the dependency
|
||||
front-end (`deps.edn` resolution, see below). A run with no `deps.edn` never
|
||||
touches the resolver.
|
||||
|
||||
Needs `jpm` and a recent Janet — developed and CI-tested against **1.41**. The
|
||||
futures and core.async layers use Janet's threaded `ev/` channels (`ev/thread`,
|
||||
`ev/thread-chan`), so older Janets may not run the full suite.
|
||||
|
||||
`jpm build` doesn't always notice source changes; run `jpm clean && jpm build`
|
||||
after editing `src/` to be sure the binaries are current. `jpm test` runs against
|
||||
the source directly, so it never goes stale.
|
||||
The bootstrap seed is **checked in**. After changing a seed source — the reader
|
||||
(`host/chez/reader.ss`), the analyzer/IR/backend (`jolt-core/jolt/*.clj`), or the
|
||||
`clojure.core` overlay (`jolt-core/clojure/core/*.clj`) — re-mint the seed with
|
||||
`make remint` (it iterates `host/chez/bootstrap.ss` to a byte-fixpoint), or
|
||||
`make selfhost` fails. Runtime-only `host/chez/*.ss` shims don't need a re-mint.
|
||||
|
||||
## How namespaces are found
|
||||
|
||||
|
|
@ -47,99 +39,74 @@ come from:
|
|||
at runtime;
|
||||
- the `:paths` option to `init` when embedding Jolt as a library.
|
||||
|
||||
If a namespace isn't found on any root, the loader falls back to the stdlib baked
|
||||
into the binary — that's how `clojure.string` and friends resolve when you run
|
||||
the binary outside the source tree.
|
||||
If a namespace isn't found on any root, the loader falls back to the stdlib in
|
||||
the overlay — that's how `clojure.string` and friends resolve when you run
|
||||
outside the source tree.
|
||||
|
||||
So you can point Jolt at a directory of Clojure source with no deps machinery at
|
||||
all:
|
||||
|
||||
```bash
|
||||
JOLT_PATH=/path/to/lib/src build/jolt myfile.clj
|
||||
JOLT_PATH=/path/to/lib/src bin/joltc run myfile.clj
|
||||
```
|
||||
|
||||
## Dependencies via deps.edn
|
||||
|
||||
`jolt` reads a `deps.edn` in the current directory, fetches its dependencies,
|
||||
and puts the resolved source directories on `JOLT_PATH` for the run. A `deps.edn`
|
||||
in the working dir is **auto-resolved** for the runnable commands (`repl`, `-m`,
|
||||
`-e`, `nrepl-server`, a `FILE`); the explicit subcommands below also work
|
||||
anywhere:
|
||||
`bin/joltc` reads a `deps.edn` in the current directory, fetches its
|
||||
dependencies, and prepends the resolved source directories to the source roots
|
||||
for the run. The CLI commands (`jolt.deps` + `jolt.main`):
|
||||
|
||||
```bash
|
||||
jolt -M:test [args] # run the :test alias's :main-opts (the usual entry)
|
||||
jolt -A:dev repl # run a command with the :dev alias's extra paths/deps
|
||||
jolt run FILE [args] # resolve, then run FILE
|
||||
jolt path # print the resolved roots (':'-joined)
|
||||
jolt tasks # list :tasks from deps.edn
|
||||
jolt task NAME [args] # run a task
|
||||
bin/joltc run -m NS [args] # resolve deps.edn, load NS, call its -main
|
||||
bin/joltc run FILE # resolve deps.edn, load a Clojure file
|
||||
bin/joltc -M:alias [args] # run the alias's :main-opts
|
||||
bin/joltc -A:alias [args] # add the alias's paths/deps, then run the rest
|
||||
bin/joltc repl # start a line REPL
|
||||
bin/joltc path # print the resolved source roots (':'-joined)
|
||||
bin/joltc <task> # run a deps.edn :tasks entry
|
||||
```
|
||||
|
||||
So, for example, to start an nREPL server that loads a project and its deps,
|
||||
add `:aliases {:nrepl {:main-opts ["nrepl-server"]}}` to `deps.edn` and run
|
||||
`jolt -M:nrepl` (or just `jolt nrepl-server`, which auto-resolves the `deps.edn`).
|
||||
|
||||
Example `deps.edn`:
|
||||
|
||||
```clojure
|
||||
{:paths ["src"]
|
||||
:deps {weavejester/medley {:git/url "https://github.com/weavejester/medley"
|
||||
:git/tag "1.0.0"}
|
||||
:git/sha "<full-sha>"}
|
||||
my/helpers {:local/root "../helpers"}}}
|
||||
```
|
||||
|
||||
```bash
|
||||
jolt run -m myapp.main
|
||||
bin/joltc run -m myapp.main
|
||||
```
|
||||
|
||||
### What's supported
|
||||
|
||||
- **git deps** — `{:git/url … :git/tag …}` or `{:git/url … :git/sha …}` (use a
|
||||
full SHA; `git fetch` can't resolve a short one). Transitive deps from each
|
||||
dependency's own `deps.edn` are resolved too.
|
||||
- **git deps** — `{:git/url … :git/sha …}` (use a full SHA; `git fetch` can't
|
||||
resolve a short one), with an optional `:deps/root` for a subdirectory.
|
||||
Transitive deps from each dependency's own `deps.edn` are resolved too.
|
||||
- **local deps** — `{:local/root "../path"}`.
|
||||
- The project's own `:paths` (default `["src"]`) are included.
|
||||
- **aliases** — `:aliases {:dev {:extra-paths ["dev"] :extra-deps {…}
|
||||
:main-opts ["-e" "…"]}}`, selected with `-A:dev` (or several: `-A:dev:test`).
|
||||
`:extra-paths`/`:extra-deps` accumulate across selected aliases;
|
||||
`:main-opts` is last-wins and runs via `-M:alias`.
|
||||
- **user config** — a `deps.edn` under `$JOLT_CONFIG` (else
|
||||
`$XDG_CONFIG_HOME/jolt`, else `~/.jolt`) merges beneath the project's, the
|
||||
way `~/.clojure/deps.edn` does: `:deps`/`:aliases`/`:tasks` merge per key
|
||||
with the project winning.
|
||||
- **tasks** — `:tasks {clean "rm -rf target" test {:doc "run the suite"
|
||||
:main-opts ["-e" "(run-tests)"]}}`. A string task is a shell command; a map
|
||||
task runs jolt with its `:main-opts`. `jolt tasks` lists, `jolt task NAME`
|
||||
runs.
|
||||
- **tasks** — `:tasks {clean "rm -rf target" test {:main-opts ["-m" "…"]}}`.
|
||||
A string task is a shell command; a map task runs jolt with its `:main-opts`.
|
||||
Run one with `bin/joltc <taskname>`.
|
||||
|
||||
Conflicts resolve the tools.deps way: resolution is breadth-first, so a
|
||||
top-level coordinate always beats a transitive one for the same lib, and
|
||||
conflicting coordinates print a warning naming both.
|
||||
Resolution is breadth-first, so a top-level coordinate always beats a transitive
|
||||
one for the same lib.
|
||||
|
||||
Git clones land in a global, sha-immutable cache shared across projects —
|
||||
`$JOLT_GITLIBS`, else `<config-dir>/gitlibs` (the `~/.gitlibs` model). The
|
||||
resolved roots are cached per project in `.cpcache/jolt-deps.jdn`, keyed on a
|
||||
hash of the project `deps.edn` + the user `deps.edn` + the selected aliases.
|
||||
`$JOLT_GITLIBS`, else `~/.jolt/gitlibs`.
|
||||
|
||||
### What's not
|
||||
|
||||
- **No Maven.** `:mvn/version` deps are ignored — git and local only.
|
||||
- **No Maven.** `:mvn/version` deps are skipped with a warning — git and local
|
||||
only.
|
||||
- **Pure `clj`/`cljc` only.** A library that needs the JVM (Java interop, host
|
||||
classes) or a `clojure.core` feature Jolt doesn't implement will fail to load
|
||||
or fail at a call. Coverage is per-function: a namespace can load with most
|
||||
functions working and a few not.
|
||||
|
||||
### Bundling into one file
|
||||
|
||||
`jolt uberscript OUT.clj -m NS` bundles `NS` and every namespace it requires —
|
||||
your code plus its dependencies — into a single `.clj` in dependency order,
|
||||
ending with a call to `NS/-main`. Run it from a project dir and the `deps.edn`
|
||||
is resolved first, so dependency namespaces are on the path to bundle. The
|
||||
result runs on a plain `jolt` with no `JOLT_PATH`, no deps fetched, and no jpm:
|
||||
|
||||
```bash
|
||||
jolt uberscript app.clj -m myapp.main
|
||||
jolt app.clj arg1 arg2
|
||||
```
|
||||
|
||||
See [`tools-deps.md`](tools-deps.md) for the design rationale.
|
||||
|
|
|
|||
|
|
@ -1,48 +0,0 @@
|
|||
# Chez port — Phase 0 results (jolt-cf1q.1)
|
||||
|
||||
De-risk + contract harness. Done; all gates green. Decisions feed Phases 1–3.
|
||||
|
||||
## 0a — value model (`host/chez/values.ss`, `test/chez/values-test.ss`)
|
||||
Jolt value layer on Chez: nil sentinel (distinct from `#f`/`'()`), interned
|
||||
keywords (NUL-separated intern key, no ns/name collision), ns+meta symbols,
|
||||
exactness-aware `jolt=` ((= 1 1.0) is false), and a `jolt-hash` consistent with
|
||||
it (non-finite-float safe). Chez's numeric tower IS Clojure's — ratios + bignums
|
||||
come free. **37/37 tests.**
|
||||
|
||||
## 0b — host-neutral contract gate (`test/chez/`)
|
||||
The spec corpus is data, so one contract gates every host. Extracted 2655
|
||||
`[label expected actual]` cases from `test/spec/*.janet` into `corpus.edn` (valid
|
||||
as BOTH EDN and Janet data). `run-corpus.janet` drives ANY jolt binary (pluggable
|
||||
`JOLT_BIN`) at the CLI boundary, one fresh subprocess per case. Baseline vs Janet
|
||||
`build/jolt` (compile mode, the port's target mode): **2641/2655**, 14 known CLI
|
||||
divergences allowlisted (interpret-vs-compile leniency + invoke-collection-as-fn,
|
||||
several non-canonical vs JVM anyway). **The gate fails only on NEW divergences** —
|
||||
exactly what we want pointed at the Chez host in Phase 1+.
|
||||
|
||||
## 0c — persistent-collection perf (`spike/chez/collections-experiment.ss`)
|
||||
The shim-vs-self-hosted decision for collections. Map-churn workload from
|
||||
`bench/collections.clj` (30000 assoc/get over 4096 keys), correct result (30000):
|
||||
|
||||
| | mean | vs Janet | vs native ceiling |
|
||||
|---|--:|--:|--:|
|
||||
| Janet jolt HAMT | 258.6 ms | 1× | — |
|
||||
| Chez persistent HAMT (hand-Scheme) | 6.3 ms (opt3) | **~41×** | ~15× |
|
||||
| Chez native hashtable (mutable) | 0.43 ms | ~600× | 1× |
|
||||
|
||||
**Decision: self-host the persistent collections in Clojure (jolt-core).** A
|
||||
persistent HAMT on the Chez substrate is ~41× faster than Janet's, so the
|
||||
substrate is not the bottleneck; a compiled-Clojure HAMT should land near the
|
||||
hand-Scheme one (cf. the mandelbrot finding that Chez compiles emitted code to
|
||||
the native ceiling). The ~15× gap to mutable-native is the inherent persistence
|
||||
cost (node-copy per assoc), identical in kind to JVM Clojure, and closes with
|
||||
transients/editable nodes when needed. Keep a Scheme-shim HAMT as fallback ONLY
|
||||
if Phase 2 shows the compiled-Clojure version underperforms.
|
||||
|
||||
Caveats (spike scope): the experiment uses integer-key-as-hash (shallow,
|
||||
collision-free trie) and `merge-leaves` lacks real collision nodes — fine for the
|
||||
substrate-speed question; the real RT needs `jolt-hash` + collision handling.
|
||||
|
||||
## Net
|
||||
Substrate speed (compute + collections), value model, and the parity gate are all
|
||||
validated and green. Phase 1 can bootstrap the real pipeline against a known,
|
||||
enforceable contract.
|
||||
|
|
@ -1,198 +0,0 @@
|
|||
# Re-hosting Jolt on Chez Scheme — phased plan
|
||||
|
||||
Decision (2026-06-17): port jolt's runtime substrate from Janet to Chez Scheme
|
||||
(cisco/ChezScheme). The spike (`spike/chez/RESULTS.md`) validated the thesis:
|
||||
the compute-substrate ceiling is ~12-47x faster than Janet (mandelbrot
|
||||
166->13.4ms matching jolt's own C-codegen result; fib 246->5.2ms), Chez's
|
||||
native compiler reaches that at runtime with the REPL intact, size stays
|
||||
single-digit MB, and the one regression (memory baseline ~2.5x) is opt-in
|
||||
tunable via WPO + stripped boot + AOT-under-petite.
|
||||
|
||||
This plan is built around two north stars beyond raw speed:
|
||||
|
||||
1. **Zero Janet — Chez is the sole substrate** (revised 2026-06-18). The goal is
|
||||
not a minimal Janet shim that coexists with Chez; it is to rip Janet out
|
||||
entirely and rely on Chez going forward. Two things must move off Janet: the
|
||||
**runtime** (a hand-written Chez Scheme RT replaces the Janet value layer /
|
||||
vars / evaluator) and the **compiler**. The compiler is the subtle part: the
|
||||
analyzer/IR are already portable Clojure, but today they *execute on the Janet
|
||||
host*, and the IR->Scheme emitter + driver are *Janet code* (`host/chez/
|
||||
emit.janet`), so the current `clojure.core` prelude is a Janet cross-compile.
|
||||
The end state requires Chez-jolt to run the analyzer itself and the emitter to
|
||||
become portable Clojure — so Chez-jolt compiles its own `clojure.core` AND the
|
||||
analyzer from source, with no Janet in the loop (the bootstrap fixpoint). Then
|
||||
both `src/jolt/*.janet` and `host/chez/*.janet` are deleted. Every line that
|
||||
stays in Scheme is hand-written Chez RT, not Janet; the forcing function for
|
||||
jolt-tzo/uqi/lcn still applies (push logic into `jolt-core/`).
|
||||
2. **Tests are the contract.** The spec/conformance corpus is host-neutral data
|
||||
(`[name expected-clj actual-clj]` triples compared via jolt's own `=`). It is
|
||||
the acceptance gate for "the port is correct" — Chez-jolt must pass the same
|
||||
corpus the Janet host passes, with no regression to the clojure-test-suite
|
||||
baseline.
|
||||
|
||||
## The Chez host RT vs portable Clojure (target end-state)
|
||||
|
||||
What MUST be hand-written Chez Scheme (the irreducible primitive layer the
|
||||
self-hosted core rests on) vs what MOVES into portable Clojure. Nothing here is
|
||||
Janet — this is the split *after* Janet is gone:
|
||||
|
||||
### Stays in Scheme (the Chez host RT)
|
||||
- **Value primitives that can't bottom out in Clojure without circularity:**
|
||||
the `nil` sentinel (distinct from `#f` and `'()` — the classic Lisp-on-Lisp
|
||||
trap), keyword/symbol records (Clojure symbols carry ns + meta), char/string
|
||||
bridging. NOTE: Chez's **numeric tower is a windfall** — int/float/ratio/
|
||||
bignum are native, so jolt gets exact ratios + bignums for free (Janet lacked
|
||||
them).
|
||||
- **Mutable cell / box + array primitives** the self-hosted RT builds on (var
|
||||
roots, transients, HAMT node arrays).
|
||||
- **A type-tag / record primitive** for deftype/protocol dispatch (Chez records).
|
||||
- **`host/compile`** — eval a backend-emitted Scheme form to a procedure. On Chez
|
||||
this is literally `eval`/`compile`. Trivial. This is the whole backend
|
||||
host-dependency.
|
||||
- **FFI / interop bridge** (foreign-procedure) for host interop calls.
|
||||
- **Persistent-collection hot nodes** — ONLY IF the Clojure-on-Chez version
|
||||
(Phase 0c) doesn't hold perf. Open question, decided by measurement.
|
||||
|
||||
### Moves into portable Clojure (`jolt-core/`)
|
||||
- **The reader** (text -> forms). CLJS self-hosts its reader; ours can too. ~33KB
|
||||
of Janet leaves the host. Not hot.
|
||||
- **Analyzer + IR + passes** — already portable Clojure source; the change is
|
||||
that they must EXECUTE on Chez-jolt, not on the Janet host (Phase 3).
|
||||
- **The backend emitter** — today it is Janet (`host/chez/emit.janet`); its LOGIC
|
||||
becomes portable Clojure (`jolt.backend-scheme`) that emits Scheme forms as
|
||||
data, so it runs on Chez. Only `host/compile` (Chez `eval`) crosses the seam.
|
||||
- **macros + clojure.core** — finish the jolt-uqi/tzo migration (most already
|
||||
Clojure).
|
||||
- **Protocol/multimethod dispatch logic** — over the host tag primitive.
|
||||
- **Persistent collections** — candidate (Phase 0c), perf-permitting.
|
||||
|
||||
### Dropped entirely
|
||||
- **The tree-walking interpreter** (`eval_base/eval_runtime/eval_special/
|
||||
eval_resolve`, ~140KB Janet). On Chez, native `compile` is always present and
|
||||
cheap, so the compile-only path can cover every form — no `jolt/uncompilable`
|
||||
fallback needed. The interpreter's role as the correctness *oracle* transfers
|
||||
to the spec corpus + JVM Clojure (the real reference), which is strictly
|
||||
better. This is the single largest shim reduction and the biggest open risk;
|
||||
Phase 1 validates that compile-only is total before we commit to the drop.
|
||||
|
||||
## Test contract strategy
|
||||
|
||||
- **Corpus is the contract.** The 44 `test/spec/*.janet` files, the conformance
|
||||
cases, `clojure-test-suite`, and `clojure-stdlib-suite` are host-neutral: pure
|
||||
Clojure source + expected values. Extract the triples into a runner that can
|
||||
target an arbitrary `jolt` binary (subprocess at the Clojure boundary).
|
||||
- **Parity gate.** Chez-jolt must pass the same corpus as Janet-jolt; the
|
||||
clojure-test-suite baseline is the bar (raise it when it rises, never lower).
|
||||
- **Oracle shift.** Today's 3-mode conformance (interpret/compile/self-host)
|
||||
loses the "interpret" leg when the interpreter is dropped; the golden expected
|
||||
values + JVM Clojure become the oracle. Keep the frozen expected values.
|
||||
- **Dual-run during migration.** Run BOTH hosts against the corpus until Chez
|
||||
reaches parity, then retire the Janet host.
|
||||
|
||||
## Phases (-> beads epic)
|
||||
|
||||
**Phase 0 — Foundations & contract harness** (de-risk; no jolt pipeline yet)
|
||||
- 0a. Chez RT value model: nil sentinel, keyword/symbol records, numeric-tower
|
||||
mapping, `=`/hashing. Resolve the nil/`'()`/`#f` representation up front.
|
||||
- 0b. Host-neutral test-contract runner: extract the spec/conformance corpus to
|
||||
drive an arbitrary jolt binary; stand up the parity-gate machinery.
|
||||
- 0c. Persistent-collection perf experiment: HAMT/PV in Clojure-on-Chez vs
|
||||
Scheme-native — the data that decides what stays shim vs self-hosted.
|
||||
|
||||
**Phase 1 — Minimal Chez kernel + real-pipeline bootstrap**
|
||||
- Scheme shim (value layer, var/ns cells, `host/compile`, cenv impl).
|
||||
- `jolt.backend` Scheme-emit target for the IR the analyzer already produces.
|
||||
- Bootstrap jolt-core (ir/analyzer) on Chez; compile + run `(+ 1 2)` -> fib ->
|
||||
mandelbrot through the REAL pipeline. Gate: compute benches run end-to-end and
|
||||
hit ~the spike ceiling; confirm compile-only is total (no fallback needed).
|
||||
|
||||
**Phase 2 — clojure.core to spec parity**
|
||||
- Bring up persistent collections (per 0c) + seq/coll/print/refs/io tiers over
|
||||
the Chez RT. Gate: spec + conformance + clojure-test-suite parity with the
|
||||
Janet baseline.
|
||||
|
||||
**Phase 3 — Self-host the compiler on Chez** (the no-Janet spine)
|
||||
- Rewrite the IR->Scheme emitter from Janet (`host/chez/emit.janet` + `driver.
|
||||
janet`) into portable Clojure in jolt-core (a `jolt.backend-scheme` target);
|
||||
folds jolt-lcn. Move the reader into jolt-core. Stand up Chez compile-from-
|
||||
source: Chez-jolt reads the `.clj` tiers, runs the analyzer *executing on Chez*
|
||||
(not Janet), emits Scheme, evals — replacing the Janet cross-compile of the
|
||||
prelude. Bootstrap fixpoint: Chez-jolt compiles `clojure.core` AND the analyzer
|
||||
from source with no Janet in the loop; verify stage2==stage3 emitted forms.
|
||||
Drop the tree-walking interpreter. Continue core-* leaf migration (jolt-uqi/
|
||||
ded/tzo). Gate: Chez-jolt builds itself from source, full corpus parity holds,
|
||||
zero Janet invoked.
|
||||
|
||||
**Phase 4 — Deployment & optimization modes** (the "optimize specific cases" lever)
|
||||
- Wire `JOLT_WHOLE_PROGRAM`/direct-link to emit specialized Scheme (fl*/fx*),
|
||||
feeding Chez `compile-whole-program`. `jolt build`: WPO + strip-fasl +
|
||||
AOT-under-petite + heap tuning -> small fast binary (jolt-0w9u reframed).
|
||||
Rebuild fibers/async on call/cc + threads. Gate: full bench suite incl.
|
||||
collections/binary-trees (the GC axes); size + memory measured vs spike
|
||||
baseline.
|
||||
|
||||
**Phase 5 — Delete the Janet host** (single substrate)
|
||||
- Chez self-hosts + parity + perf confirmed -> delete both `src/jolt/*.janet`
|
||||
(the seed) and `host/chez/*.janet` (the Janet emitter/driver, now superseded by
|
||||
the Clojure `jolt.backend-scheme`). Chez is the only substrate; no alternate
|
||||
host retained. jolt-core unchanged. Oracle stays the spec corpus + JVM Clojure.
|
||||
Net: one host, no Janet, no cross-compile.
|
||||
|
||||
## Host interop & the examples acceptance corpus
|
||||
|
||||
The `../examples` repo holds real jolt apps with real git deps + C interop; they
|
||||
are the **end-to-end acceptance gate** complementing the unit-level spec corpus.
|
||||
The port must account for jolt's interop surface, which is layered:
|
||||
|
||||
- **`janet.*` bridge** — the general Clojure->Janet escape hatch (`janet/get`,
|
||||
`janet/struct`, `janet.ev/sleep`, `janet.net/close`, `janet.spork.http/*`),
|
||||
used mostly in demo glue. DECIDED (2026-06-17): rename the bridge to a
|
||||
**neutral `host.*`** namespace (not `janet.*`/`chez.*`), so app interop code is
|
||||
host-portable; each host implements `host.*` over its own FFI (Janet FFI today,
|
||||
Chez `foreign-procedure` tomorrow). The legacy `janet.*` aliases stay as
|
||||
deprecated shims during migration.
|
||||
- **FFI-backed shim libraries** — the genuine C interop. `jolt-lang/http-client`
|
||||
implements `java.net` + TLS + gzip as host shims **over Janet FFI**, backing
|
||||
clj-http-lite's `:clj` branch. On Chez these are reimplemented over Chez FFI
|
||||
(libcurl/openssl/zlib or native sockets); the `java.*`-facing API is unchanged.
|
||||
Also in scope: `jolt-lang/db`, `jolt-lang/logging`, and `jolt-lang/router`
|
||||
(mirrors `reitit.Trie`; verify whether it carries native code).
|
||||
- **`:jpm/module` Janet-native deps** — `spork/http` (server), `ring-janet-
|
||||
adapter`. No Chez equivalent: provide a Scheme/Chez-FFI HTTP server or treat
|
||||
these as test-only fixtures.
|
||||
- **Native-library dependencies in deps.edn** (DECIDED 2026-06-17). C-interop
|
||||
shims need shared libraries (libcurl, openssl/libssl, zlib) that CANNOT be
|
||||
pulled from git like Clojure libs. Add a `:native` dep form so a project can
|
||||
*declare* what it needs, e.g.
|
||||
`{:native/lib "curl" :native/min-version "7.0" :native/header "curl/curl.h"}`.
|
||||
The resolver doesn't fetch them, but: (a) it surfaces the requirement to the
|
||||
user (and can suggest the install command per platform), and (b) the
|
||||
loader/`foreign-procedure` layer probes for the `.so`/`.dylib` at load and, if
|
||||
absent, raises a precise error — "missing native library: libcurl (declared by
|
||||
jolt-lang/http-client); install with `brew install curl`" — instead of a raw
|
||||
dlopen failure. Symmetric with `:jpm/module`'s "verify importable, hint to
|
||||
install" pattern.
|
||||
- **Host-neutral pieces that port for free** — Java-class mirror shims (pure
|
||||
Clojure) and `JOLT_FEATURES` reader conditionals.
|
||||
|
||||
Sequencing: the interop bridge mechanism is part of the Phase-1 shim, but the
|
||||
FFI-backed libraries land in **Phase 4** (downstream of core parity), validated
|
||||
by running the examples end-to-end. Tracked as a dedicated epic child.
|
||||
|
||||
## Beads reconciliation
|
||||
|
||||
- **Closed (obsolete — Janet bytecode-VM / cgen / Janet-dispatch mechanisms Chez
|
||||
replaces wholesale):** jolt-ffn (epic, already concluded flat), jolt-5vsp.1,
|
||||
jolt-qx70, jolt-l1l4 (cgen), jolt-cm7t, jolt-fw2 (Janet dispatch substrate),
|
||||
jolt-pria (Janet ctx cold-build startup).
|
||||
- **Recommend close (confirm) — Janet constant-factor passes Chez's JIT/GC/WPO
|
||||
subsume:** jolt-826, jolt-27w, jolt-t6r, jolt-8flj, jolt-3ko, jolt-t34,
|
||||
jolt-u1f. (jolt-ffn's own STATUS says the gap is "generic-runtime overhead…
|
||||
the JVM erases via JIT + inline caching + unboxing" — exactly Chez's job.)
|
||||
- **Reframed under the Chez epic:** jolt-0w9u + .1 -> Phase 4 deploy mode +
|
||||
closed-world audit; jolt-1r86 -> Phase 0b/4 bench validation; jolt-lcn /
|
||||
jolt-uqi / jolt-tzo / jolt-ded / jolt-brh / jolt-7dl -> Phase 2/3 self-hosting,
|
||||
now targeting Chez; jolt-5vsp (foundational-runtime epic) -> parent, the Chez
|
||||
port is its realization.
|
||||
- **Keep host-agnostic:** deps beads (jolt-x4o, jolt-xkd, jolt-pnje, jolt-vley);
|
||||
correctness bug jolt-jk23.
|
||||
|
|
@ -1,198 +0,0 @@
|
|||
# Foundational Runtime Epic — Handoff
|
||||
|
||||
**Epic:** jolt-5vsp · **Predecessor:** jolt-ffn (targeted specialization — concluded)
|
||||
**Date:** 2026-06-16
|
||||
|
||||
This is a cold-start handoff. Read it top to bottom before touching code. Its
|
||||
whole point is to keep the fresh session from re-running the experiments that
|
||||
already came back flat, and to start from the one measurement that actually
|
||||
tells us where to invest.
|
||||
|
||||
## Why this epic exists
|
||||
|
||||
The targeted-specialization epic (jolt-ffn) tried to close jolt's constant-factor
|
||||
gap vs JVM Clojure with per-form compiler passes. Three independent attempts all
|
||||
came back flat:
|
||||
|
||||
| Attempt | Bead | Result |
|
||||
|---|---|---|
|
||||
| Record field-read guard removal (bare field reads) | jolt-3ko | ~3% on dispatch (shipped #141 — kept for correctness, not speed) |
|
||||
| Protocol inline cache (runtime, per-method) | jolt-ez5h | ~0% — the per-dispatch gen-check exactly cancels the find-protocol-method saving; `find` was never the bottleneck |
|
||||
| Record-ctor descriptor-baking (fewer allocs/record) | jolt-p7fo | flat on binary-trees + broke the gate; reverted |
|
||||
|
||||
The conclusion: **the gap is structural to jolt-on-Janet, not a missing
|
||||
optimization.** Targeted passes remove only the cheap parts; the structural floor
|
||||
remains.
|
||||
|
||||
## The scorecard (jolt / JVM Clojure)
|
||||
|
||||
Regenerate any time with `JVM=1 bench/run.sh` (the absolute-reference mode).
|
||||
|
||||
| Axis | Bench | jolt/JVM |
|
||||
|---|---|---|
|
||||
| Pure float compute | `mandelbrot` | **~15× ← THE FLOOR** |
|
||||
| Persistent collections (HAMT) | `collections` | ~28× |
|
||||
| Recursion (call + arith) | `fib` | ~37× |
|
||||
| Megamorphic dispatch | `dispatch` | ~76× |
|
||||
| Monomorphic dispatch | `mono-dispatch` | ~109× |
|
||||
| Allocation / GC | `binary-trees` | ~314× (≈150× at depth 10) |
|
||||
|
||||
`mandelbrot` is the floor: pure tight arithmetic loops — no dispatch, no
|
||||
allocation, no collections — and native arith already fires (jolt-3pl). So ~15×
|
||||
is what jolt's *execution substrate* costs on the simplest possible workload.
|
||||
Every other axis adds structural overhead **on top** of that floor.
|
||||
|
||||
**Machine caveat:** the dev machine swaps heavily (~13 GB). Alloc-heavy benches
|
||||
(`binary-trees`, `collections`) inflate badly; light benches (`mandelbrot`,
|
||||
`fib`, `dispatch`) are trustworthy. Get absolute alloc numbers on a clean machine.
|
||||
|
||||
## The four structural walls
|
||||
|
||||
1. **Bytecode-VM execution.** jolt's backend emits **Janet** (a register-bytecode
|
||||
VM) and runs it on the Janet interpreter loop — no JIT, no native code. Every
|
||||
op is bytecode dispatch. This is the `mandelbrot` 15× floor.
|
||||
2. **Mark-sweep GC.** Janet's GC scans all live objects each cycle (no
|
||||
generations). Live-data + alloc-heavy workloads (`binary-trees` retains the
|
||||
tree) pay O(live) per GC. The JVM's generational GC makes young-object churn
|
||||
nearly free.
|
||||
3. **Indirect calls.** Protocol dispatch and fn calls go through indirection
|
||||
(closures, the protocol registry). The JVM inlines/devirtualizes. jolt's
|
||||
devirt (jolt-41m) only fires on *statically*-proven monomorphic sites;
|
||||
`reduce`/`mapv` over a collection doesn't give that proof, so the common
|
||||
runtime-monomorphic case pays full dispatch (that's why `mono-dispatch` is
|
||||
*worse* than megamorphic — the JVM inline-caches it to near-free, jolt doesn't).
|
||||
4. **Boxed / generic representations.** Records are tuples `[descriptor field…]`;
|
||||
field access goes through a tag guard unless the type is proven. Generic ops
|
||||
carry runtime type checks. (Open question: are Janet *numbers* boxed? Verify
|
||||
in the spike — it decides whether unboxing is a lever or already done.)
|
||||
|
||||
## Foundational levers (ranked)
|
||||
|
||||
1. **Native codegen — emit C, not Janet bytecode.** The Stalin approach. Compile
|
||||
jolt IR → C → machine code via the system compiler. The *only* lever that
|
||||
moves the 15× compute floor; could approach C/JVM speed on compute-bound code.
|
||||
Massive (a new backend). Plausible incremental shape: a jolt-IR→C compiler for
|
||||
*hot* fns with a fallback to the existing bytecode path for unsupported forms —
|
||||
mirroring today's interpret/compile hybrid. Needs to confirm Janet's C-API /
|
||||
native-module story can be targeted incrementally.
|
||||
2. **Structural GC-pressure reduction.** Value-type small records (avoid heap),
|
||||
transient/editable-node hot paths (RFC 0003 future work — pvec/phm/sorted are
|
||||
now tries/HAMT/RB, so O(1) `transient`/`persistent!` via editable nodes is
|
||||
open). Helps the alloc-bound axes (`binary-trees`, `collections`). Does **not**
|
||||
touch the compute floor.
|
||||
3. **Deeper devirt + body inline.** Propagate element/return types so devirt
|
||||
fires on runtime-monomorphic collections, then inline the method body
|
||||
(jolt-4x9 element types + jolt-t6r). Helps dispatch. Bounded ceiling (still
|
||||
bytecode underneath).
|
||||
|
||||
## STATUS (2026-06-16) — lever 1 (native codegen) built and working
|
||||
|
||||
The spike ran and lever 1 is now implemented. Full writeups:
|
||||
`docs/foundational-runtime-spike-results.md` (floor localization) and
|
||||
`docs/foundational-runtime-lever1-native-codegen.md` (native codegen).
|
||||
|
||||
Done (all merged to main, PRs #143–#148):
|
||||
- **Floor localized:** the 15.4× decomposes into a **Janet-VM floor ≈10.8× JVM**
|
||||
(only native codegen moves it) + a **jolt loop-lowering ≈1.43×** (cheap backend
|
||||
win, bead **jolt-v28u**). Janet numbers are already unboxed (not a lever).
|
||||
- **Native codegen (jolt-ihdp, CLOSED):** `src/jolt/cgen.janet` translates
|
||||
numeric-leaf fns (numeric in/out, native-op arithmetic + loop/recur/if/let/do)
|
||||
to C. Wired into the backend `:def` emit under **`JOLT_CGEN=1`** (opt-in). The
|
||||
`.so` is content-addressed + cached. **mandelbrot 224ms → 12.4ms (~18×)**,
|
||||
beats JVM. Leaf-first falls out free (callers stay bytecode, call native fn).
|
||||
- **Build-time AOT (jolt-a7ds, partial):** `:cgen-collect?` records leaf fns at
|
||||
build, `aot-build` compiles them into one `.so` + manifest; `:cgen-prebuilt` +
|
||||
`load-aot` install them at deploy with **no cc** (proven with cc off PATH).
|
||||
|
||||
Open work under epic jolt-5vsp:
|
||||
- **jolt-a7ds** — fuse the prebuilt `.so` + manifest into the `jpm` exe for a
|
||||
literal single binary (+ a `jolt cgen-build -m app` CLI). The heaviest piece;
|
||||
into jpm executable-build, not the compiler.
|
||||
- **jolt-v28u** — `while`-loop lowering for tail `recur` (cheap ~30%, independent
|
||||
of cgen; helps ALL loops, not just cgen candidates).
|
||||
- **jolt-l1l4** — widen cgen numeric grammar (mod/rem/bit-ops/min/max, mixed fns).
|
||||
- **jolt-qx70** — hot-fn auto-detection (drop the global `JOLT_CGEN` knob).
|
||||
- Lever 2 (GC-pressure) and lever 3 (deeper devirt) — untouched; see below.
|
||||
|
||||
The original spike instructions are preserved below for context.
|
||||
|
||||
**Localize the 15× floor.** Build three `mandelbrot` implementations and compare:
|
||||
|
||||
- **jolt-compiled** `mandelbrot` (already in `bench/mandelbrot.clj`),
|
||||
- **hand-written Janet** `mandelbrot` (the same nested loop, idiomatic Janet —
|
||||
write it directly, no jolt),
|
||||
- **JVM Clojure** `mandelbrot`.
|
||||
|
||||
Two ratios fall out:
|
||||
|
||||
- **jolt-emitted-Janet vs hand-Janet** → how much overhead jolt's *backend* adds
|
||||
over optimal Janet. To see jolt's emitted Janet, use the backend emit path
|
||||
(`backend/emit-ir` on the analyzed `run`/`count-point` fns) — note `:arities`
|
||||
etc. are jolt pvecs, so introspection is awkward; easier to read the emitted
|
||||
Janet via the compile path or just A/B the timings.
|
||||
- **hand-Janet vs JVM** → the Janet VM's own floor.
|
||||
|
||||
Decision:
|
||||
|
||||
- If **hand-Janet ≈ jolt** and hand-Janet is ~15× JVM → the floor is **Janet's
|
||||
bytecode VM**. Native codegen (lever 1) is the only fix. Commit to the spike of
|
||||
a jolt-IR→C path for one hot fn and measure.
|
||||
- If **jolt ≫ hand-Janet** → jolt's backend emits suboptimal Janet; there's
|
||||
headroom in the **backend** (cheaper, no new runtime). Find what it emits that
|
||||
hand-Janet doesn't.
|
||||
|
||||
Also measure the **GC share** on `binary-trees` (Janet GC stats around the run —
|
||||
`(gccollect)` / `gcinterval`, or count allocations) to size lever 2 honestly.
|
||||
|
||||
## Key files / mechanisms
|
||||
|
||||
- **Backend (IR → Janet emit):** `src/jolt/backend.janet`. `native-ops` (~L322)
|
||||
emits native Janet arith; `emit-ir` (~L674) runs passes then emits. A native-C
|
||||
backend would branch here.
|
||||
- **Passes / inference:** `jolt-core/jolt/passes.clj` (`run-passes`),
|
||||
`jolt-core/jolt/passes/types.clj` (inference; the `:fn` branch ~L527 now seeds
|
||||
^Record param hints — #141), `jolt-core/jolt/passes/inline.clj`
|
||||
(scalar-replace, `ctor-shape`).
|
||||
- **Record representation:** `src/jolt/types_protocols.janet` — `make-record`
|
||||
(~L145, the ~5-alloc/record path), `record-shape-for` (~L139, rebuilds its
|
||||
cache key every call), `record-tag`. Records are tuples `[descriptor field…]`.
|
||||
- **Dispatch + ctors:** `src/jolt/eval_runtime.janet` —
|
||||
`protocol-dispatch-impl` (~L62), `make-deftype-ctor-impl` (~L382).
|
||||
- **Config knobs:** `src/jolt/config.janet` — `JOLT_DIRECT_LINK`,
|
||||
`JOLT_WHOLE_PROGRAM`, `JOLT_OPTIMIZE`, the `ctx-shaping-env-vars` list (any new
|
||||
ctx-shaping env var MUST be added there and to `image-cache-path`).
|
||||
- **Self-hosting design:** `docs/self-hosting-compiler.md` (the kernel/value-layer
|
||||
boundary), `docs/rfc/0003-transients.md` (editable-node future work).
|
||||
|
||||
## How to build, run, measure
|
||||
|
||||
```sh
|
||||
jpm build # build/jolt (ctx baked, ~20ms startup); from-source is ~8s cold
|
||||
export PATH="$PWD/build:$PATH"
|
||||
bench/run.sh # jolt only, WP on
|
||||
JVM=1 bench/run.sh # jolt vs JVM scorecard (needs `clojure` on PATH)
|
||||
bench/run.sh mandelbrot 400 # one bench, custom size
|
||||
JOLT_WHOLE_PROGRAM=0 bench/run.sh # measure what WP buys
|
||||
```
|
||||
|
||||
Gate: `jpm build; janet run-tests.janet` (parallel, ~100s; `JOLT_TEST_JOBS`
|
||||
overrides). Bench memory hygiene (`bd memories bench-isolation-gotcha`): never run
|
||||
a perf matrix while other CPU work runs — it starves later configs and produces
|
||||
bogus numbers. Sandwich A/B/A.
|
||||
|
||||
## What NOT to repeat (already flat — see beads for detail)
|
||||
|
||||
- Runtime protocol inline cache (jolt-ez5h): gen-check cancels the saving.
|
||||
- Field-read guard removal as a *speed* play (jolt-3ko): ~3%; machinery dominates.
|
||||
(The #141 change is kept for correctness + the `with-meta`-on-symbols fix.)
|
||||
- `make-record` descriptor-baking (jolt-p7fo): flat — `binary-trees` is dominated
|
||||
by the live retained tree + GC, not the short-lived intermediate allocs.
|
||||
|
||||
## Open questions for the spike
|
||||
|
||||
- Are Janet numbers boxed? (Lever or already done.)
|
||||
- Does Janet expose a native-module / C-codegen path jolt can target incrementally
|
||||
(hot fns → C, rest → bytecode)?
|
||||
- What fraction of `binary-trees` is GC vs execution?
|
||||
- Is there a cheaper record representation (Janet struct vs tuple-with-descriptor)
|
||||
that lowers field-read + alloc cost without a new backend?
|
||||
|
|
@ -1,173 +0,0 @@
|
|||
# Lever 1 — Native codegen (jolt-IR → C): feasibility spike
|
||||
|
||||
**Epic:** jolt-5vsp · **Date:** 2026-06-16
|
||||
**Predecessor:** the localization spike (`docs/foundational-runtime-spike-results.md`)
|
||||
showed the 15.4× mandelbrot floor is ~70% Janet-VM floor (only native codegen
|
||||
moves it) + ~30% loop-lowering (cheap backend fix, jolt-v28u). This spike probes
|
||||
**lever 1's ceiling and the incremental hot-fn-in-C strategy** before committing
|
||||
to a backend.
|
||||
|
||||
All legs return the identical result (3288753 at n=200). Numbers are means of 3
|
||||
after warmup; the dev machine swaps, so treat these as orders-of-magnitude (the
|
||||
≈ vs JVM call is robust; ±2ms is noise).
|
||||
|
||||
## The native-C ceiling — it beats JVM
|
||||
|
||||
Native mandelbrot built as a Janet native module (`spike/native/mandel.c`):
|
||||
|
||||
| Leg | mean | vs jolt (219ms) | vs JVM (14.2ms) |
|
||||
|---|---|---|---|
|
||||
| **native-C whole run** (pure C, no Janet in loop) | **~10–12 ms** | **~18–22× faster** | **faster than JVM** |
|
||||
| Janet loop → C hot-fn (forward crossing) | ~11–13 ms | ~18× faster | ≈ JVM |
|
||||
| C loop → `janet_call` bytecode (reverse crossing) | ~152 ms | ~no better | ~11× slower |
|
||||
| *(reference)* jolt-compiled | 219 ms | — | 15.4× |
|
||||
| *(reference)* JVM Clojure | 14.2 ms | — | 1.0× |
|
||||
|
||||
**Verdict: lever 1 is validated and its ceiling is excellent.** Compiling the hot
|
||||
compute path to C makes it ~18–22× faster than today's jolt and *edges out JVM
|
||||
Clojure* — native code has no VM-dispatch floor at all. This is the only lever
|
||||
that touches the ~10.8× Janet-VM floor, and the payoff is the full gap.
|
||||
|
||||
## The crossing-direction rule (the key strategic finding)
|
||||
|
||||
The boundary cost is wildly asymmetric:
|
||||
|
||||
- **Forward (bytecode → C): nearly free.** A Janet bytecode loop calling a C
|
||||
hot-fn n² (=40 000) times runs at ~11–13 ms — within ~15% of pure C. So you can
|
||||
compile just the *inner* hot fn to C and capture ~95% of the win while the outer
|
||||
loop stays bytecode. **Incremental adoption works.**
|
||||
- **Reverse (C → `janet_call` → bytecode): ~3.5 µs/call.** A C fn calling a
|
||||
bytecode helper per iteration runs at ~152 ms — *no better than jolt today*. The
|
||||
`janet_call` cost (entering the VM/fiber per call) dominates.
|
||||
|
||||
**Design constraint → compile leaf-first / whole-hot-cluster.** A fn is a
|
||||
profitable C-compilation candidate only if its hot path calls **nothing that stays
|
||||
in bytecode** — only primitives or other C-compiled fns. Cross the boundary only at
|
||||
*cold* edges. For mandelbrot, `count-point` is a leaf (calls only arithmetic
|
||||
primitives) → the ideal first target; compiling it alone captures the win
|
||||
(forward crossing), but a half-compiled hybrid that `janet_call`s back per
|
||||
iteration buys nothing.
|
||||
|
||||
## The dynamic-compile path works (no jpm needed)
|
||||
|
||||
jolt's compile model is dynamic (analyze → IR → Janet → eval at runtime). Native
|
||||
codegen fits the same shape: a `.so` compiled with a **plain `cc` invocation**
|
||||
(no jpm/project.janet) loads at runtime via `require` and runs at full native
|
||||
speed (verified: `run-c(200)` correct, 13.5 ms cold).
|
||||
|
||||
```
|
||||
cc -shared -fPIC -O2 -I/opt/homebrew/include -undefined dynamic_lookup \
|
||||
mandel.c -o mandel.so # macOS; Linux drops -undefined dynamic_lookup
|
||||
(require "path/to/mandel") # loads at runtime, cfunctions callable
|
||||
```
|
||||
|
||||
So the native tier mirrors today's interpret/compile hybrid: emit C for a hot
|
||||
fn → shell to `cc` → `require` the `.so` → bytecode callers call into it via the
|
||||
(cheap, forward) native-module call path. Caching keyed by fn-source-hash mirrors
|
||||
the existing ctx image cache.
|
||||
|
||||
## Toolchain confirmed (this machine)
|
||||
|
||||
- `janet.h` present (`/opt/homebrew/include/janet.h`, Janet 1.41.2).
|
||||
- `jpm declare-native` builds a `.so` cleanly.
|
||||
- Direct `cc` (no jpm) builds a loadable `.so`.
|
||||
- C API used: `janet_getnumber/getinteger`, `janet_wrap_number`, `janet_fixarity`,
|
||||
`janet_getfunction`, `janet_call`, `janet_cfuns`, `JANET_MODULE_ENTRY`.
|
||||
|
||||
## Status: wired into the compile path (JOLT_CGEN, opt-in)
|
||||
|
||||
`src/jolt/cgen.janet` (IR→C translator) is wired into the backend's `:def` emit
|
||||
via `cgen-root`, gated behind **`JOLT_CGEN=1`** (off by default; needs
|
||||
direct-linking). When on, a plain defn of a numeric-leaf fn is compiled to C at
|
||||
def time and the cfunction installed as the var root — so direct-linked callers
|
||||
embed native code. The fn is NOT inline-stashed when cgen fires (callers must
|
||||
call the C fn, not inline the bytecode body). `^:redef`/`^:dynamic` defns stay
|
||||
bytecode.
|
||||
|
||||
The leaf-first rule emerges for free: `run` calls `count-point` (a user var, not
|
||||
a native-op), so `run` isn't a numeric leaf and stays bytecode — calling the
|
||||
native `count-point` over the cheap forward crossing.
|
||||
|
||||
**Measured end-to-end (`jolt -m mandelbrot 200`): 224 ms → 12.4 ms, ~18×**, with
|
||||
the correct result — matching the spike's native-C ceiling. The default gate
|
||||
(cgen off) is unchanged. Tests: `test/integration/cgen-pipeline-test.janet`.
|
||||
|
||||
Known limitation: building *core* with `JOLT_CGEN=1` would try to cgen core
|
||||
numeric-leaf fns into the cached ctx image, where embedded cfunctions may not
|
||||
serialize — keep cgen for app/user code until image-cache interaction is handled.
|
||||
|
||||
## Build-time AOT: native speed without a toolchain on the target (jolt-a7ds)
|
||||
|
||||
The JIT path above runs `cc` at runtime. The AOT path moves compilation to build
|
||||
time so the deploy target needs no `cc`/`janet.h`:
|
||||
|
||||
- **Build phase** (`:cgen-collect?`, needs cc): loading the app records every
|
||||
numeric-leaf defn's IR; `cgen/aot-build` compiles them all into ONE native
|
||||
module (`gen-c-module`) and `write-manifest` persists `{sopath, [{ns name sym}]}`.
|
||||
- **Deploy phase** (`:cgen-prebuilt`, NO cc): `cgen/load-aot` loads the prebuilt
|
||||
`.so` (via the `native` builtin — no compiler) into a qname→cfunction map; the
|
||||
backend's `:def` hook installs each as the var root with the same timing as the
|
||||
JIT path, so callers direct-link to native code.
|
||||
|
||||
**Proven** (`spike/native/aot-demo.janet`, two processes): build with cc, then
|
||||
deploy with `cc` removed from PATH → `count-point` is still native, mandelbrot =
|
||||
3288753 at **12.4 ms** (full 18×). Test: `test/integration/cgen-aot-test.janet`.
|
||||
|
||||
This removes the runtime-toolchain dependency — the core of the deployment story.
|
||||
|
||||
### The literal single binary (`jolt cgen-build`, done)
|
||||
|
||||
`src/jolt/cgen_build.janet` + the `jolt cgen-build -m NS -o OUT` CLI fuse the
|
||||
native code into the executable, so an app ships as ONE static file — no sidecar
|
||||
`.so`, no toolchain to run. The driver:
|
||||
|
||||
1. loads the app with `:cgen-collect?` to get its numeric-leaf fns + the source
|
||||
files loaded (the uberscript-style bundle);
|
||||
2. emits `cg.c` (one native module of those fns via `cgen/gen-c-module`) + a
|
||||
positional manifest;
|
||||
3. stages a build dir: `src`/`jolt-core` symlinks into the jolt tree, `cg.c`, the
|
||||
app bundle, and an entry that bakes the runtime, installs the native fns as var
|
||||
roots (`:cgen-prebuilt`), and runs `-main`;
|
||||
4. runs `jpm build` there — `declare-native` builds `cg.a`, `declare-executable`
|
||||
static-links it into the final exe (jpm's `create-executable` marshals the
|
||||
module's cfunctions and calls its static entry at startup).
|
||||
|
||||
Build needs `cc` + `jpm`; the result needs neither. Proven end-to-end:
|
||||
`test/integration/cgen-build-test.janet` builds the mandelbrot fixture, runs it
|
||||
from a clean dir with no `src/` and no `cg.so`, and gets the right total at native
|
||||
speed (the count-point leaf is the linked cfunction).
|
||||
|
||||
Build mechanics that bit (codified in `cgen_build.janet`): `stdlib_embed` slurps
|
||||
`.clj` relative to cwd, so the build runs in a dir mirroring the repo layout (the
|
||||
symlinks); jpm hardcodes `./project.janet` and sets `syspath = modpath`; the
|
||||
executable's dofile imports `cg` and static-links `cg.a`, neither ordered nor
|
||||
release-built by default, so the deps are wired explicitly; cleanup must `lstat`
|
||||
(never follow the tree symlinks). The inner build runs `--workers=1` so it doesn't
|
||||
saturate cores inside the parallel test gate.
|
||||
|
||||
Open follow-ups (filed): widen the cgen grammar (jolt-l1l4) so more of an app's
|
||||
hot fns qualify; hot-fn auto-detection (jolt-qx70) to drop the manual collect;
|
||||
DCE on the bundled source (the uberscript path already does it).
|
||||
|
||||
## Open questions for the implementation (next beads)
|
||||
|
||||
1. **IR→C for the numeric subset.** Translate jolt IR → C for proven-double
|
||||
arithmetic + tail `loop`/`recur` (count-point's shape). The native-arith type
|
||||
proof (jolt-3pl) that already gates native *Janet* arith is the same proof that
|
||||
gates C unboxing — reuse it. Start narrow: unbox doubles at entry, primitive
|
||||
ops inline, rebox at exit; bail to bytecode for any unsupported form.
|
||||
2. **Boundary policy.** Non-primitive args stay Janet values (no unbox);
|
||||
per-iteration calls allowed only to other C-compiled fns. Encode the
|
||||
leaf-first/cluster rule as the compile-candidate predicate.
|
||||
3. **Trigger + cache.** AOT at build/first-run vs lazy JIT on hot fns; `.so`
|
||||
cache keyed by source hash + flags (add to `ctx-shaping-env-vars` /
|
||||
image-cache machinery if it becomes a ctx knob).
|
||||
4. **Coverage.** Closures/upvalues, multi-arity, `recur` across the C boundary,
|
||||
portability of `cc` flags per platform.
|
||||
|
||||
## Artifacts (`spike/native/`)
|
||||
|
||||
- `mandel.c` — native mandelbrot: `run-c` (pure C), `count-point-c` (leaf cfn),
|
||||
`run-callback` (C loop → `janet_call` back, the reverse-crossing probe)
|
||||
- `project.janet` — `declare-native` build
|
||||
- `bench-native.janet` — the three-leg benchmark + harness
|
||||
|
|
@ -1,92 +0,0 @@
|
|||
# Foundational Runtime Spike — Results (the 15× floor, localized)
|
||||
|
||||
**Epic:** jolt-5vsp · **Date:** 2026-06-16
|
||||
**Spike:** the START HERE section of `docs/foundational-runtime-handoff.md` —
|
||||
jolt vs hand-written-Janet vs JVM `mandelbrot`, to localize the ~15× compute
|
||||
floor before committing to native codegen (lever 1) vs a backend fix.
|
||||
|
||||
## Setup
|
||||
|
||||
Three implementations of the same nested mandelbrot loop, all returning the
|
||||
identical result (3288753 at n=200, confirming correctness across all legs):
|
||||
|
||||
- **jolt-compiled** — `bench/mandelbrot.clj` (`jolt -m mandelbrot 200`, WP + direct-link on)
|
||||
- **hand-Janet (`while`)** — `bench/mandelbrot-hand.janet` (idiomatic Janet: `while` + `var`/`set`)
|
||||
- **JVM Clojure** — `bench/mandelbrot.clj` on the JVM
|
||||
|
||||
Plus a diagnostic fourth leg:
|
||||
|
||||
- **hand-Janet (recursive)** — `bench/mandelbrot-hand-rec.janet`: hand Janet that
|
||||
*mirrors jolt's loop lowering* (self-recursive local closure called per
|
||||
iteration), to test whether the loop lowering alone explains jolt's overhead.
|
||||
|
||||
Numbers are stable and sandwiched (A/B/A/B); machine noise < 1%.
|
||||
|
||||
## The numbers (n=200, mean of 3, after warmup)
|
||||
|
||||
| Leg | mean | × JVM |
|
||||
|---|---|---|
|
||||
| JVM Clojure | 14.2 ms | 1.0× |
|
||||
| **hand-Janet (`while`)** | **153.4 ms** | **10.8×** |
|
||||
| hand-Janet (recursive, mirrors jolt) | 215.3 ms | 15.2× |
|
||||
| **jolt-compiled** | **219.0 ms** | **15.4×** |
|
||||
|
||||
## What this localizes
|
||||
|
||||
The 15.4× floor **decomposes into two distinct layers**:
|
||||
|
||||
1. **Janet VM floor ≈ 10.8× JVM** (70% of the gap). Optimal hand-written Janet —
|
||||
pure `while` loop over unboxed doubles, zero allocation — is still ~11× slower
|
||||
than JVM Clojure. This is the cost of the Janet bytecode VM itself (no JIT, no
|
||||
native code). **Only native codegen (lever 1) can touch this.** It is the
|
||||
dominant share and validates lever 1 as the big structural lever.
|
||||
|
||||
2. **jolt backend loop-lowering ≈ 1.43× on top** (the remaining 30%). jolt is
|
||||
`219 / 153 = 1.43×` slower than optimal Janet. The diagnostic leg pins this
|
||||
*entirely* to one cause: jolt lowers every `loop`/`recur` to a **self-recursive
|
||||
local closure called once per iteration**, not a `while` loop. Hand-Janet
|
||||
written that same way (recursive leg) lands at **215 ms ≈ jolt's 219 ms** —
|
||||
so the recursive-closure lowering accounts for essentially all of jolt's
|
||||
backend overhead on pure-compute code.
|
||||
|
||||
See the emitted Janet (`bench/dump-mandelbrot-emit.janet`): `emit-loop`
|
||||
(`src/jolt/backend.janet:210`) produces
|
||||
`(do (var L nil) (set L (fn (i zr zi) … (L (+ i 1) …))) (let (…) (L …)))`
|
||||
and `emit-recur` (`:228`) produces the per-iteration call `(L …)`. It relies
|
||||
on Janet TCO for stack safety, but each iteration still pays a function
|
||||
invocation (frame setup + arg bind) that a `while` loop skips.
|
||||
|
||||
## Decision
|
||||
|
||||
The handoff posed it as binary (Janet-VM floor *or* backend headroom). It is
|
||||
**both**, now sized:
|
||||
|
||||
- **Native codegen (lever 1) is the only thing that moves the dominant ~70%.**
|
||||
Confirmed as the big lever. Pursue the incremental jolt-IR→C spike for one hot
|
||||
fn next, per the handoff.
|
||||
- **A cheap, localized ~30% win sits in the backend**, independent of any new
|
||||
runtime: lower tail-position `loop`/`recur` with scalar bindings to a Janet
|
||||
`while` + `var`/`set` instead of a recursive closure. Closes the 1.43×, taking
|
||||
`mandelbrot` from 15.4× → ~10.8× JVM. Filed separately (see epic children).
|
||||
|
||||
## Open questions answered
|
||||
|
||||
- **Are Janet numbers boxed?** No — already unboxed. The `while` leg does pure
|
||||
double arithmetic at a steady 153 ms with no allocation and no GC stutter, and
|
||||
matches the other legs bit-for-bit. Janet's `number` is an immediate IEEE
|
||||
double (stored inline in the Janet value, not heap-allocated). **Unboxing is
|
||||
not a lever; it's done.**
|
||||
- **GC share of `binary-trees`** — not measured here (the dev machine swaps
|
||||
heavily, which distorts alloc-heavy benches; the handoff flags this). Size
|
||||
lever 2 on a clean machine. The `mandelbrot` legs are alloc-free so are
|
||||
unaffected and trustworthy.
|
||||
- **Janet native-module / incremental C path** — not yet confirmed; this is the
|
||||
gating question for the lever-1 spike (hot fns → C, rest → bytecode).
|
||||
|
||||
## Artifacts (kept in `bench/`)
|
||||
|
||||
- `mandelbrot-hand.janet` — optimal `while` Janet (the Janet VM floor reference)
|
||||
- `mandelbrot-hand-rec.janet` — recursive-closure Janet (the loop-lowering diagnostic)
|
||||
- `dump-mandelbrot-emit.janet` — dumps the Janet jolt emits for the hot fns
|
||||
|
||||
The bench harness (`bench/run.sh`) ignores these (it iterates a fixed bench list).
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
# Clojure libraries known to work with Jolt
|
||||
|
||||
Libraries confirmed to load and pass their conformance checks on Jolt
|
||||
(see `test/integration/deps-conformance-test.janet` and the
|
||||
(see the [examples](https://github.com/jolt-lang/examples), e.g. the
|
||||
[ring-app example](https://github.com/jolt-lang/examples/tree/main/ring-app)).
|
||||
|
||||
* [config](https://github.com/yogthos/config)
|
||||
|
|
@ -19,7 +19,7 @@ Libraries confirmed to load and pass their conformance checks on Jolt
|
|||
* [honeysql](https://github.com/seancorfield/honeysql) — full formatter + helpers
|
||||
(select/insert/update/delete/joins/:inline), loaded unmodified from git
|
||||
* [clojure.jdbc](https://github.com/yogthos/clojure.jdbc) — as [jolt-lang/db](https://github.com/jolt-lang/db)'s
|
||||
`jdbc.core`, reimplemented over janet sqlite3/pq drivers (SQLite + PostgreSQL)
|
||||
`jdbc.core`, over the built-in SQLite access (libsqlite3 via Chez's FFI)
|
||||
* [next.jdbc](https://github.com/seancorfield/next-jdbc) — a compatibility layer in
|
||||
[jolt-lang/db](https://github.com/jolt-lang/db) (`next.jdbc`, `next.jdbc.sql`,
|
||||
`next.jdbc.prepare`, `next.jdbc.transaction`) over `jdbc.core`, for libraries
|
||||
|
|
@ -33,10 +33,7 @@ Libraries confirmed to load and pass their conformance checks on Jolt
|
|||
`logf`/`logp`, `spy`, and `enabled?` all work; output goes to stderr.
|
||||
* [migratus](https://github.com/yogthos/migratus) — database migrations; loads
|
||||
unmodified and runs filesystem SQL/EDN migrations against SQLite through the
|
||||
next.jdbc layer above. `migrate`/`rollback` round-trip end to end. Caveat:
|
||||
migration ids are 14-digit timestamps, and the janet-lang/sqlite3 driver
|
||||
currently truncates INTEGER columns to 32 bits, so completion tracking needs
|
||||
the one-line upstream fix (`sqlite3_column_int64`); ids under 2^31 work as is.
|
||||
next.jdbc layer above. `migrate`/`rollback` round-trip end to end.
|
||||
* [malli](https://github.com/metosin/malli) — data schema validation, on the
|
||||
[malli-app example](https://github.com/jolt-lang/examples/tree/main/malli-app).
|
||||
`m/validate` and `m/explain` work across the vocabulary (predicates, `:int`/
|
||||
|
|
|
|||
|
|
@ -170,9 +170,9 @@ Signature(s), since-version
|
|||
## Open questions
|
||||
|
||||
1. Numerics: the reference has longs/doubles/ratios/BigInt with promotion
|
||||
rules; CLJS has JS numbers; jolt has Janet numbers. Likely answer: specify
|
||||
an integer/float core with a host-numeric-tower extension point — needs
|
||||
its own design note in §4.
|
||||
rules; CLJS has JS numbers. Resolved: jolt carries the Scheme numeric tower
|
||||
(exact integers/bignums, exact ratios, flonum doubles), matching the
|
||||
reference's tower — see the numerics note in §4.
|
||||
2. Where do `*print-length*`-style dynamic vars land — host-dependent
|
||||
interface or portable with defaults?
|
||||
3. License/venue if the spec outgrows this repo (likely CC-BY; separate repo
|
||||
|
|
|
|||
|
|
@ -1,50 +1,49 @@
|
|||
# RFC 0003: Transients — semantics and why they live in the Janet seed
|
||||
# RFC 0003: Transients — semantics and the Chez mutable backing
|
||||
|
||||
Status: accepted (design note)
|
||||
|
||||
This note pins down what transients *are* in Jolt, where their behavior
|
||||
deviates from JVM Clojure and why, and why the transient machinery is part of
|
||||
the irreducible Janet seed rather than a candidate for the core-in-Clojure
|
||||
migration (jolt-tzo). It exists so the kernel-shrink ladder doesn't revisit
|
||||
deviates from JVM Clojure and why, and how the transient machinery is
|
||||
represented in the Chez runtime. It exists so the design doesn't revisit
|
||||
transients every round.
|
||||
|
||||
## What a transient is in Jolt
|
||||
|
||||
A transient is a tagged Janet table wrapping a *native* mutable host value
|
||||
(`core.janet`, "Transients" section):
|
||||
A transient is a Chez record (`jolt-transient`, `host/chez/transients.ss`)
|
||||
wrapping *true mutable* host backing, snapshotted to the immutable collection on
|
||||
`persistent!`. The backing is per kind:
|
||||
|
||||
- transient vector — `@{:jolt/type :jolt/transient :kind :vector :arr ARRAY}`,
|
||||
a Janet array.
|
||||
- transient map — `:kind :map :tbl TABLE`, a Janet table mapping
|
||||
`canon-key(k)` → `@[k v]`. Keying by canonical key keeps collection keys
|
||||
comparing by value across representations (`[1 2]` the pvec and `[1 2]` the
|
||||
tuple are one key), and storing the `@[k v]` pair preserves the *original*
|
||||
key for the rebuilt persistent map.
|
||||
- transient set — `:kind :set :tbl TABLE` mapping `canon-key(x)` → `x`.
|
||||
- transient vector — a growable Scheme vector (a capacity buffer plus a fill
|
||||
count `n`). `conj!`/`pop!` are in-place, amortized O(1); the buffer doubles on
|
||||
growth.
|
||||
- transient map — a Chez hashtable keyed by `key-hash` / `jolt=`
|
||||
(value-equality, nil-safe). Hashing by value keeps collection keys comparing
|
||||
across representations.
|
||||
- transient set — a Chez hashtable of elements.
|
||||
- `cow` — a copy-on-write fallback for anything else (e.g. a sorted coll).
|
||||
|
||||
`transient` accepts pvecs, mutable-build arrays, tuples (reader vectors and
|
||||
map entries — added in the seed-shrink rounds so `(into [] (first {:a 1}))`
|
||||
works through the vector fast path), sets, phms, and untagged struct maps.
|
||||
Sorted collections are rejected, as on the JVM (not editable).
|
||||
`transient` accepts pvecs, pmaps, psets, and the exotic colls handled by the
|
||||
`cow` path. Each kind copies its source into the matching mutable backing once.
|
||||
|
||||
The bang ops (`conj!`, `assoc!`, `dissoc!`, `disj!`, `pop!`) mutate that host
|
||||
value in place and return the transient — O(1) per op (amortized for array
|
||||
push). `persistent!` rebuilds a persistent value from the host value and
|
||||
invalidates the transient (`:jolt/persistent` flag; any further bang op or a
|
||||
second `persistent!` throws "Transient used after persistent! call", matching
|
||||
Clojure's invalidation contract).
|
||||
The bang ops (`conj!`, `assoc!`, `dissoc!`, `disj!`, `pop!`) mutate that backing
|
||||
in place and return the transient — O(1) per op (amortized for the vector push).
|
||||
`persistent!` snapshots a persistent value from the backing (folding the
|
||||
hashtable into a pmap/pset, handing off the buffer as a pvec) and invalidates the
|
||||
transient (the record's active flag clears; any further bang op or a second
|
||||
`persistent!` throws "transient used after persistent!", matching Clojure's
|
||||
invalidation contract).
|
||||
|
||||
Read ops work on an active transient where Clojure supports them: `get`,
|
||||
`contains?`, `count`, and `nth` (vector kind) branch on the transient tag.
|
||||
`contains?`, `count`, and `nth` (vector kind) see through the transient.
|
||||
`seq` on a transient is not supported, as in Clojure.
|
||||
|
||||
## Deviations from JVM Clojure (deliberate)
|
||||
|
||||
**O(n) edges, O(1) middle.** Clojure's `(transient v)` is O(1) — the transient
|
||||
*shares* the persistent trie and marks nodes editable; `persistent!` is O(1)
|
||||
too. Jolt's `transient` copies the source into a native array/table (O(n)) and
|
||||
`persistent!` rebuilds (O(n)). The bang ops in between are native-host O(1),
|
||||
which is *faster* per-op than trie editing. So the asymptotics of the usual
|
||||
too. Jolt's `transient` copies the source into a mutable buffer/hashtable (O(n))
|
||||
and `persistent!` snapshots back (O(n)). The bang ops in between are host-mutable
|
||||
O(1), which is *faster* per-op than trie editing. So the asymptotics of the usual
|
||||
pattern
|
||||
|
||||
(persistent! (reduce conj! (transient []) coll))
|
||||
|
|
@ -53,13 +52,12 @@ are identical (O(n) total either way) with a better constant in the loop and a
|
|||
worse constant at the two edges. The pattern transients exist for — batch
|
||||
construction — is fully served. What is NOT served is transient-editing a
|
||||
*large* collection to change a few keys: that's O(n) in Jolt vs O(log n) in
|
||||
Clojure, because `transient` flattens the pvec trie / phm HAMT into a
|
||||
native array/table and `persistent!` rebuilds them.
|
||||
Clojure, because `transient` copies the source into a growable Scheme vector /
|
||||
Chez hashtable and `persistent!` snapshots it back.
|
||||
|
||||
**No thread-ownership check.** JVM Clojure ≥1.7 also dropped the owner-thread
|
||||
assertion (for fork/join), keeping only "don't use after persistent!", which
|
||||
Jolt enforces. Jolt code is fiber-concurrent; when real OS-thread futures land
|
||||
(jolt-ejx), a transient handed across threads is a data race exactly as in
|
||||
Jolt enforces. A transient handed across threads is a data race exactly as in
|
||||
Clojure — documented, not checked, same as the JVM.
|
||||
|
||||
**`(conj!)` / `(conj! t)` arities** follow Clojure's transducer-era contract:
|
||||
|
|
@ -68,51 +66,43 @@ zero args makes a fresh `(transient [])`, one arg returns it untouched.
|
|||
lenient kvs walk of Jolt's `assoc`.
|
||||
|
||||
**No transient sorted variants** — same as Clojure. One leniency: Clojure
|
||||
throws on `(transient '(1))`, but Jolt's lists are Janet arrays underneath and
|
||||
fall into the mutable-build branch, yielding a transient *vector*. Harmless
|
||||
(the result of `persistent!` is a vector, never silently a list) but
|
||||
non-Clojure; tighten if it ever bites.
|
||||
throws on `(transient '(1))`, but Jolt routes a list through the `cow` fallback
|
||||
path, yielding a transient. Harmless but non-Clojure; tighten if it ever
|
||||
bites.
|
||||
|
||||
## Why transients stay in the Janet seed
|
||||
## Why transients live in the host
|
||||
|
||||
The migration ladder (jolt-tzo) moves anything expressible as *pure Clojure
|
||||
over existing primitives* out of the seed. Transients fail that test on three
|
||||
Transients are part of the value/representation layer in the Chez runtime
|
||||
(`host/chez/transients.ss`), not the portable `clojure.core` overlay, on three
|
||||
grounds:
|
||||
|
||||
1. **They are the mutation kernel.** A transient's entire value is direct
|
||||
mutation of a host array/table. The overlay's only mutation seam is
|
||||
`jolt.host/ref-put!` (a single table-put). Re-expressing `tr-conj!` etc. in
|
||||
Clojure would mean either growing the host surface one-for-one
|
||||
(`host-array-push!`, `host-table-put!`, …, i.e. moving the same code behind
|
||||
more indirection) or simulating mutation over persistent values (defeating
|
||||
the point of transients). Either way the Janet line count moves, it doesn't
|
||||
shrink.
|
||||
mutation of a host buffer/hashtable. The overlay has no mutation seam of its
|
||||
own. Re-expressing the bang ops in Clojure would mean either growing the host
|
||||
surface one-for-one (a host-vector-push, a host-hashtable-put, …, i.e. moving
|
||||
the same code behind more indirection) or simulating mutation over persistent
|
||||
values (defeating the point of transients).
|
||||
|
||||
2. **They sit under the seed's own dispatch.** `conj`/`assoc`/`get`/`count`/
|
||||
`contains?` in the seed branch on the transient tag. Hoisting the transient
|
||||
ops above that dispatch (the hierarchy-port pattern of lazily-resolved
|
||||
overlay vars) would put an interpreted/compiled-Clojure call inside the
|
||||
hottest native paths for no semantic gain — transients have no semantics to
|
||||
*fix* (unlike hierarchy, which had real correctness gaps).
|
||||
2. **They sit under the collection dispatch.** `conj`/`assoc`/`get`/`count`/
|
||||
`contains?` see through a transient. Hoisting the transient ops above that
|
||||
dispatch would put a compiled-Clojure call inside the hottest paths for no
|
||||
semantic gain — transients have no semantics to *fix*.
|
||||
|
||||
3. **The value layer is declared irreducible.** The self-hosting design doc
|
||||
(docs/self-hosting-compiler.md, "The kernel") keeps the value/representation
|
||||
layer — persistent collections and, with them, their mutable scratch
|
||||
counterparts — in the host. Transients are representation, not library.
|
||||
3. **The value layer is the host's job.** The persistent collections and, with
|
||||
them, their mutable scratch counterparts, live in the Chez runtime alongside
|
||||
the value model. Transients are representation, not library.
|
||||
|
||||
What CAN move (and mostly has): anything *derived* — e.g. `into`'s
|
||||
transient-using fast path, or future `update!`-style conveniences — is plain
|
||||
Clojure over `transient`/bang-ops/`persistent!` and belongs in the overlay
|
||||
tiers as ordinary migration batches.
|
||||
What lives in the overlay: anything *derived* — e.g. `into`'s transient-using
|
||||
fast path, or `update!`-style conveniences — is plain Clojure over
|
||||
`transient`/bang-ops/`persistent!`.
|
||||
|
||||
## Future work
|
||||
|
||||
- pvec is already a 32-way trie with structural sharing (pv.janet), so
|
||||
Clojure-style O(1) `transient`/`persistent!` via editable nodes is a real
|
||||
option for vectors — an internal change behind the same surface, not a
|
||||
semantics change. phm is now a HAMT with structural sharing too (jolt-684u),
|
||||
and sorted maps/sets are a red-black tree (jolt-0hbr), so the same editable-
|
||||
node trick is open for those as well — the transient surface here is still the
|
||||
copy-to-native-table flatten.
|
||||
- The persistent map/set are a bitmap HAMT with structural sharing
|
||||
(`host/chez/collections.ss`), so Clojure-style O(1) `transient`/`persistent!`
|
||||
via editable nodes is a real option there — an internal change behind the same
|
||||
surface, not a semantics change. The persistent vector is a flat
|
||||
copy-on-write Scheme vector rather than a trie, so the transient surface for
|
||||
it stays the copy-to-growable-vector path.
|
||||
- `transient?` (Jolt extension, useful in tests) stays; Clojure has no public
|
||||
predicate, so it must not leak into portability-sensitive code.
|
||||
|
|
|
|||
|
|
@ -11,26 +11,19 @@ measured effect, so later work does not relitigate it.
|
|||
## Background: why the lookup carries a guard
|
||||
|
||||
A Jolt map value has several runtime representations (see RFC on collections and
|
||||
`src/jolt/core.janet`): a Janet struct for a small all-scalar-key literal map, a
|
||||
persistent hash map (a table tagged `:jolt/type :jolt/phm`) when a key is a
|
||||
collection or a value is nil, plus sorted maps, transients, and record/deftype
|
||||
instances. A record instance is a Janet table tagged `:jolt/deftype` but, like a
|
||||
struct, it carries no `:jolt/type`, so a raw Janet `(get inst :field)` reads its
|
||||
fields directly.
|
||||
`host/chez/collections.ss`): a persistent hash map (a bitmap HAMT) for the
|
||||
general case, plus sorted maps, transients, and record/deftype instances. A
|
||||
record instance is a Chez record (`jrec`) whose fields are read directly off the
|
||||
record's storage, while a HAMT lookup runs the full `jolt=`/`jolt-hash`-keyed
|
||||
collection path.
|
||||
|
||||
A constant-keyword lookup `(:k m)` compiles to a guarded form:
|
||||
|
||||
```janet
|
||||
(if (get m :jolt/type) (core-get m k) (get m k))
|
||||
```
|
||||
|
||||
The guard is one opcode. A non-nil `:jolt/type` routes phm/sorted/transient/
|
||||
lazy-seq values to `core-get`'s full semantics; everything else (structs,
|
||||
records, nil, scalars) takes the bare Janet `get`, which matches `core-get` for
|
||||
keyword keys. The guard is correct and cheap, but on a struct it is a second
|
||||
`get`: profiling the ray tracer (a naive all-maps program) found keyword lookups
|
||||
are about half of a render, and the guard is the only avoidable part of each
|
||||
one. A bare get is roughly 20ns where the guarded form is roughly 36ns.
|
||||
A constant-keyword lookup `(:k m)` compiles to a guarded form: it inspects the
|
||||
subject's representation and routes a HAMT/sorted/transient/lazy-seq value to the
|
||||
full `jolt-get` semantics, while a record/raw-get-safe value takes the direct
|
||||
field read, which matches `jolt-get` for keyword keys. The guard is correct and
|
||||
cheap, but on a raw-get-safe value it is wasted work: profiling the ray tracer (a
|
||||
naive all-maps program) found keyword lookups are about half of a render, and the
|
||||
guard is the only avoidable part of each one.
|
||||
|
||||
Dropping the guard is only safe when the subject is known to be a plain
|
||||
struct/record rather than a tagged collection. Jolt does not infer that
|
||||
|
|
@ -59,27 +52,27 @@ optimization.
|
|||
## How it flows
|
||||
|
||||
The reader already keeps `^hint` metadata on the binding symbol and is otherwise
|
||||
transparent (`reader.janet`, `meta-form->map`). The change threads that fact to
|
||||
the lookup site:
|
||||
transparent (`host/chez/reader.ss`). The change threads that fact to the lookup
|
||||
site:
|
||||
|
||||
1. The analyzer (`jolt-core/jolt/analyzer.clj`) records a `:struct` hint per
|
||||
local in its env when a param or `let` binding carries `^:struct` or a
|
||||
record-type tag, and attaches `:hint :struct` to that local's `:local` IR
|
||||
node. Resolving a record-type tag uses a new host contract function
|
||||
`record-type?` (`src/jolt/host_iface.janet`), which checks for the `->Name`
|
||||
constructor.
|
||||
2. The back end (`emit-kw-lookup` in `src/jolt/backend.janet`) emits the bare get
|
||||
node. Resolving a record-type tag uses the host contract function
|
||||
`record-type?` (`jolt.host`, backed by `host/chez/host-contract.ss`), which
|
||||
checks for the `->Name` constructor.
|
||||
2. The back end (`jolt-core/jolt/backend_scheme.clj`) emits the direct field read
|
||||
when the lookup subject is a `:local` carrying the hint, and the guarded form
|
||||
otherwise. The unhinted path is byte-identical to before.
|
||||
otherwise. The unhinted path is identical to before.
|
||||
3. The inline pass (`jolt-core/jolt/passes.clj`) propagates the hint: when it
|
||||
binds a non-trivial call argument to a fresh local, it carries the called
|
||||
function's parameter hint onto that local, so lookups inside the spliced body
|
||||
keep the bare path. Without this, inlining a hinted function would erase the
|
||||
keep the direct path. Without this, inlining a hinted function would erase the
|
||||
benefit, because the hinted parameter is replaced by an unhinted temporary.
|
||||
|
||||
The same machinery covers both `(:k m)` and `(get m :k [default])` when the key
|
||||
is a constant keyword. A `get` with a variable, numeric, or string key falls
|
||||
through to `core-get` unchanged.
|
||||
through to `jolt-get` unchanged.
|
||||
|
||||
## Record hints across namespaces, and as inference seeds
|
||||
|
||||
|
|
@ -98,15 +91,14 @@ the function where the hot reads actually happen.
|
|||
|
||||
**It resolves across namespaces.** A hint may name a record defined in another
|
||||
namespace, in either spelling — `^Vec3` where the type is `:refer`-ed, or
|
||||
`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key` in
|
||||
`src/jolt/host_iface.janet`, backed by `record-hint-ctor-key` in
|
||||
`src/jolt/evaluator.janet`) runs against the *compile* namespace and maps the
|
||||
type to its home constructor key through a constructor-value index — keyed by the
|
||||
constructor value, not a var's namespace, so a `:refer`-interned var (whose
|
||||
namespace is the referring one) still resolves home. The reader keeps a tag's
|
||||
namespace qualifier (`^v/Vec3` → `"v/Vec3"`, not `"Vec3"`) so the aliased
|
||||
spelling has something to resolve. Both `defrecord` field hints and function
|
||||
parameter hints use this resolution.
|
||||
`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key`,
|
||||
a `jolt.host` contract function backed by `host/chez/host-contract.ss`) runs
|
||||
against the *compile* namespace and maps the type to its home constructor key
|
||||
through a constructor-value index — keyed by the constructor value, not a var's
|
||||
namespace, so a `:refer`-interned var (whose namespace is the referring one)
|
||||
still resolves home. The reader keeps a tag's namespace qualifier (`^v/Vec3` →
|
||||
`"v/Vec3"`, not `"Vec3"`) so the aliased spelling has something to resolve. Both
|
||||
`defrecord` field hints and function parameter hints use this resolution.
|
||||
|
||||
## Soundness and the checked mode
|
||||
|
||||
|
|
@ -120,8 +112,8 @@ To make a lie visible without taxing the fast path, `JOLT_CHECK_HINTS=1` keeps
|
|||
the guard but throws on the tagged arm with a message naming the local and key:
|
||||
|
||||
```
|
||||
type hint violated on `m`: (:a m) — value carries :jolt/type
|
||||
(a phm/sorted/transient/lazy-seq), not the plain struct/record the
|
||||
type hint violated on `m`: (:a m) — value is a
|
||||
phm/sorted/transient/lazy-seq, not the plain struct/record the
|
||||
^:struct/^Record hint asserts
|
||||
```
|
||||
|
||||
|
|
@ -134,7 +126,7 @@ off). The flag is part of the image-cache fingerprint.
|
|||
Type hints parse in every position Clojure accepts them and are inert except for
|
||||
the optimization above. This matches Clojure's "parse and otherwise do nothing"
|
||||
model, with the difference that Clojure additionally uses hints to avoid
|
||||
reflection and select primitive arithmetic, which do not apply to a Janet host.
|
||||
reflection and select primitive arithmetic, which do not apply to the Chez host.
|
||||
|
||||
## Measured effect
|
||||
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ A type `T` is one of:
|
|||
`:nonnil` for "provably not nil and not false", which is what the struct-vs-phm
|
||||
decision needs; see below.)
|
||||
- `:nil`.
|
||||
- `{:struct {field -> T}}` — a raw-get-safe map (Janet struct or record) whose
|
||||
- `{:struct {field -> T}}` — a raw-get-safe map (a record) whose
|
||||
field `k` has type `(fields k)` or `:any` if absent. The degenerate
|
||||
`{:struct {}}` is "a struct, fields unknown" and replaces today's
|
||||
`:struct-map`.
|
||||
|
|
@ -65,7 +65,7 @@ A type `T` is one of:
|
|||
|
||||
Types are immutable values comparable by structural equality, exactly like the
|
||||
current `{:vec ELEM}` representation, so they flow across the portable
|
||||
inference and the Janet orchestrator unchanged.
|
||||
inference and the host unchanged.
|
||||
|
||||
### Join (least upper bound)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,80 +1,45 @@
|
|||
# Seed ↔ Overlay Registry
|
||||
|
||||
Jolt is "Clojure on Janet": a shrinking **Janet seed** (`src/jolt/*.janet`)
|
||||
hosts a **Clojure overlay** (`jolt-core/clojure/core/NN-*.clj`). Both define
|
||||
`clojure.core`-facing functions, and for a handful of names *both* tiers carry a
|
||||
definition. Which copy is authoritative has been tribal knowledge. This document
|
||||
is the single source of truth; `test/unit/seed-overlay-registry-test.janet` is a
|
||||
build-time drift check that fails if reality diverges from what is written here.
|
||||
Jolt is Clojure on Chez Scheme. `clojure.core` is built from two tiers that both
|
||||
define `clojure.core`-facing vars, and for a handful of names *both* tiers carry
|
||||
a definition. This document records how the two tiers relate and which copy is
|
||||
authoritative.
|
||||
|
||||
## The registration mechanism
|
||||
## The two tiers
|
||||
|
||||
Seed core functions are named with a `core-` prefix (`core-into`, `core-conj`,
|
||||
`core-transduce`) and registered into the `clojure.core` namespace by the
|
||||
`core-bindings` table in `src/jolt/core.janet`. Each entry maps a **public
|
||||
Clojure name** (the string key) to a seed function value:
|
||||
- **Native shims** (`host/chez/natives-*.ss`) bind a set of `clojure.core` vars
|
||||
directly to Scheme runtime values via `def-var!` — collection constructors,
|
||||
seq fns, numeric/string ops, and so on. These cover names the overlay assumes
|
||||
exist as bare `clojure.core` vars but does not define itself.
|
||||
- **The Clojure overlay** (`jolt-core/clojure/core/NN-*.clj`) defines the rest of
|
||||
`clojure.core` in dependency-ordered tiers, loaded in order: `00-syntax`,
|
||||
`00-kernel`, `10-seq`, `20-coll`, `25-sorted`, `30-macros`, `40-lazy`, `50-io`.
|
||||
|
||||
```janet
|
||||
(def- core-bindings
|
||||
@{"into" core-into
|
||||
"reduce" core-reduce
|
||||
...})
|
||||
```
|
||||
|
||||
`init-core!` (`src/jolt/core.janet`) interns every pair into `clojure.core`.
|
||||
The overlay tiers load afterwards (`api.janet`: 00-syntax, 00-kernel, 10-seq,
|
||||
20-coll, 25-sorted, 30-macros, 40-lazy, 50-io). When an overlay tier `(defn X …)`
|
||||
for a name that `core-bindings` already registered, the **overlay def shadows the
|
||||
seed binding** — the seed `core-X` then survives only if some other seed code
|
||||
still calls it directly.
|
||||
The overlay loads after the native shims. When an overlay tier `(defn X …)` for a
|
||||
name a native shim already bound, the **overlay def shadows the native binding** —
|
||||
user code sees the overlay copy. The native binding then survives only if some
|
||||
other native/runtime code still calls the Scheme value directly.
|
||||
|
||||
So a name's *home* is determined by two facts:
|
||||
|
||||
1. is it a key in `core-bindings`? (registered ⇒ the seed `core-X` is reachable)
|
||||
2. does an overlay tier `(defn X …)`? (defined ⇒ the overlay copy shadows)
|
||||
1. is it bound by a native shim? (the Scheme value is reachable from the runtime)
|
||||
2. does an overlay tier `(defn X …)`? (the overlay copy is what user code sees)
|
||||
|
||||
## Dispatch-only seed helpers: the `__` prefix
|
||||
## The compiled seed
|
||||
|
||||
Seed functions that are **not** public Clojure vars but must be reachable by
|
||||
name from compiled/overlay code (compiler hooks, macro-expansion targets) are
|
||||
registered under a `__`-prefixed key — e.g. `"__sq1"`, `"__write"`,
|
||||
`"__bit-and"`, `"__jdbc-conn-raw"`. The `__` prefix is unreadable as a
|
||||
user-level symbol, so these never collide with or masquerade as public API. When
|
||||
you add a dispatch-only hook, give it a `__` key; do not register it under a bare
|
||||
name.
|
||||
`clojure.core` is compiled ahead of time into the checked-in seed
|
||||
(`host/chez/seed/{prelude,image}.ss`) as Scheme `def-var!` forms. The seed's
|
||||
source twin is the overlay (`jolt-core/clojure/core/*.clj` plus the stdlib
|
||||
namespaces under `src/jolt/clojure/`); `host/chez/emit-image.ss` re-emits the
|
||||
prelude from those sources on Chez. The build is a byte-fixpoint: rebuilding from
|
||||
an up-to-date seed reproduces it exactly.
|
||||
|
||||
## Dispatch twins
|
||||
## Consistency guard
|
||||
|
||||
A **twin** is a name with *both* a seed `core-X` defn and an overlay `(defn X …)`.
|
||||
There are exactly five. Each seed site carries a greppable `SEED-TWIN:` comment.
|
||||
|
||||
| name | overlay (authoritative public) | seed copy (`core-X`) | registered? | role of the seed copy |
|
||||
|---------------|----------------------------------------|--------------------------------------|-------------|------------------------|
|
||||
| `char?` | `20-coll.clj` `char?` | `core_types.janet` `core-char?` | no | internal type dispatch |
|
||||
| `sorted-map?` | `25-sorted.clj` `sorted-map?` | `core_types.janet` `core-sorted-map?`| no | internal dispatch (sorted-op) |
|
||||
| `sorted-set?` | `25-sorted.clj` `sorted-set?` | `core_types.janet` `core-sorted-set?`| no | internal dispatch |
|
||||
| `sorted?` | `25-sorted.clj` `sorted?` | `core_types.janet` `core-sorted?` | no | internal dispatch |
|
||||
| `transduce` | `20-coll.clj` `transduce` | `core_coll.janet` `core-transduce` | no | internal helper for `core-into` only |
|
||||
|
||||
None of the five is registered in `core-bindings`: the overlay copy is the public
|
||||
one, and the seed copy is reached only by other *seed* code (so editing the seed
|
||||
copy alone will not change what user code sees — change both, or move the logic).
|
||||
|
||||
## The surprising asymmetry: `into` vs `transduce`
|
||||
|
||||
`into` and `reduce` are **seed-public**: registered in `core-bindings`, and the
|
||||
overlay deliberately does *not* redefine them (they sit on the perf wall — see
|
||||
the "into stays in the seed" note in `20-coll.clj`). `transduce`, by contrast, is
|
||||
**overlay-public**: the overlay `transduce` is the real one, and `core-transduce`
|
||||
remains only because `core-into` calls it directly. So two functions that read as
|
||||
a matched pair have opposite homes. That asymmetry is intentional and is the
|
||||
reason this registry exists.
|
||||
|
||||
## Drift check
|
||||
|
||||
`test/unit/seed-overlay-registry-test.janet` recomputes the twin set from source
|
||||
(names with both a seed `core-X` defn and an overlay `defn X`) and asserts it
|
||||
equals the five above. It also asserts none of the twins is registered in
|
||||
`core-bindings`, and that every non-`__` `core-bindings` key is a plausible
|
||||
public name (no accidental `__`-less dispatch helper). If you add, remove, or
|
||||
re-home a twin, update this table and that test together.
|
||||
There is no separate drift-check test for the registry. The self-hosting
|
||||
fixpoint is the guard: after changing a seed source (a core tier, the compiler
|
||||
namespaces, the host contract, the reader, or `emit-image.ss`) you must re-mint
|
||||
the seed (`make remint`), and `make selfhost` fails if the checked-in seed and
|
||||
its sources have drifted. So if the overlay's shadowing relationship changes, the
|
||||
re-minted prelude changes with it, and the fixpoint check keeps source and seed
|
||||
in agreement.
|
||||
|
|
|
|||
|
|
@ -1,138 +0,0 @@
|
|||
# Self-hosting architecture: portable jolt-core over a host runtime
|
||||
|
||||
Design for splitting Jolt into a **portable Clojure-in-Clojure core** and a
|
||||
**host runtime** (Janet today, another runtime tomorrow), so the language is
|
||||
truly self-hosted and `jolt-core` can be lifted out and re-hosted.
|
||||
|
||||
This is the design that must be right *before* writing the compiler in Clojure —
|
||||
see [[self-hosting-compiler]] for the staged plan it plugs into.
|
||||
|
||||
## What "truly self-hosted + portable" requires
|
||||
|
||||
Two independent properties:
|
||||
|
||||
1. **Self-hosted** — the compiler and most of `clojure.core` are written in
|
||||
Clojure and compiled by Jolt itself.
|
||||
2. **Portable** — that Clojure code (`jolt-core`) depends only on a small,
|
||||
explicit **host contract**, never on Janet directly. Re-hosting means
|
||||
implementing the contract for a new runtime; `jolt-core` is reused verbatim.
|
||||
|
||||
The enemy is `jolt-core` calling `janet/tuple`, `make-vec`, `ns-find`, etc.
|
||||
directly — that welds it to Janet. Every host dependency must go through the
|
||||
contract.
|
||||
|
||||
## Prior art (the seam everyone uses)
|
||||
|
||||
- **Clojure (JVM).** `clojure.lang.*` (Java) is the host: `RT`/`Numbers` runtime
|
||||
helpers, the `Compiler` (form → JVM bytecode), persistent data structures,
|
||||
`Var`/`Namespace`, the reader. `clojure/core.clj` is the language, in Clojure.
|
||||
Seam: ~20 primitive special forms + `RT` static methods. Everything else is
|
||||
Clojure.
|
||||
- **ClojureScript (self-hosted).** Two portable passes — `cljs.analyzer`
|
||||
(form → AST **as data**, reading a **compiler-state map** of
|
||||
namespaces/defs/macros, *not* host objects) and `cljs.compiler` (AST → JS, the
|
||||
host-specific back end). `cljs.core` is Clojure compiled to JS. Platform splits
|
||||
live in `.cljc` reader conditionals. This is the closest model to what we want:
|
||||
**the analyzer is host-agnostic; only the back end and the runtime are
|
||||
host-specific.**
|
||||
- **Nanopass / Guile Tree-IL.** A high-level IR is the portability seam; multiple
|
||||
back ends consume it.
|
||||
- **ClojureCLR / ClojureDart / jank.** Same shape every time: portable analyzer +
|
||||
host back end + host runtime.
|
||||
|
||||
The invariant across all of them: **the IR (analyzer output) and a small runtime
|
||||
protocol are the contract; the front end is portable, the back end and runtime
|
||||
are per-host.**
|
||||
|
||||
## Decisions (locked)
|
||||
|
||||
- **Seam = a minimal host protocol.** `jolt-core` calls a small documented set of
|
||||
host fns (in ns `jolt.host`): `resolve-sym`, `macro?`, `macroexpand-1`,
|
||||
`current-ns`, `intern!`, plus the `RT` primitives. Each host provides `jolt.host`
|
||||
(+ RT). Re-hosting = reimplement that handful of fns. The protocol *is* the
|
||||
boundary; `jolt-core` never touches Janet directly.
|
||||
- **Physical split now.** Portable Clojure lives under `jolt-core/` (a new source
|
||||
root, embedded into the binary like the rest of the stdlib); host Janet code for
|
||||
the new pipeline under `host/janet/`. Legacy host modules under `src/jolt/*.janet`
|
||||
are the existing Janet host and get relocated under `host/janet/` in a later
|
||||
mechanical pass (tracked) — not moved big-bang now, to keep the suite green.
|
||||
|
||||
## The Jolt split
|
||||
|
||||
```
|
||||
jolt-core/ PORTABLE Clojure — no Janet. Depends only on the contract.
|
||||
ir the IR spec (data shapes the analyzer emits)
|
||||
analyzer form -> IR (macroexpands; resolves via host protocol)
|
||||
macros when/cond/->/defn/... (the macro library, in Clojure)
|
||||
core clojure.core fns expressible in Clojure, over RT primitives
|
||||
|
||||
host/janet/ THE HOST — Janet. Implements the contract.
|
||||
reader text -> jolt forms
|
||||
rt data structures + RT primitive fns (cons/first/+/get/apply…)
|
||||
backend IR -> Janet forms -> Janet compile -> bytecode (the emitter)
|
||||
cenv the compile-time host protocol impl (resolve/macro?/intern)
|
||||
bootstrap load jolt-core, wire analyzer+backend into the loader
|
||||
interop janet.* bridge
|
||||
```
|
||||
|
||||
Two contracts cross the seam:
|
||||
|
||||
### 1. The IR (analyzer → back end)
|
||||
The existing `:op`-tagged AST, made **host-neutral**:
|
||||
- `{:op :const :val v}`, `:if`, `:do`, `:let`, `:fn` (arities), `:invoke`,
|
||||
`:vector`/`:map`/`:set`, `:quote`, `:throw`/`:try`, `:loop`/`:recur`.
|
||||
- **Globals reference vars by NAME, not by host cell:**
|
||||
`{:op :var :ns "clojure.core" :name "map"}`. (compiler.janet today embeds the
|
||||
Janet var cell as a constant — that's a host leak and breaks AOT. Name-based
|
||||
refs are both portable and AOT-friendly; the back end resolves the cell.)
|
||||
- No embedded host function values. Calls to runtime primitives are
|
||||
`{:op :rt :name "cons"}` resolved by the back end to the host's RT fn.
|
||||
|
||||
### 2. The host contract (two protocols)
|
||||
- **Compile-time (`cenv`)** — what the analyzer needs from the host while
|
||||
analyzing: `(current-ns)`, `(resolve-sym sym) -> {:kind :var|:macro|:local|:special|:host, :ns, :name}`,
|
||||
`(macroexpand-1 form)`, `(intern! ns sym meta)`. The analyzer calls only these;
|
||||
it never touches Janet ns/var tables. (CLJS keeps this as pure data; we use a
|
||||
small protocol — a minimal, documented boundary — because Jolt already has live
|
||||
ns/var objects. The protocol *is* the seam.)
|
||||
- **Runtime (`RT`)** — the primitive fns emitted code and `jolt-core` call by
|
||||
stable name: arithmetic/compare, `cons/first/rest/seq/conj/get/assoc/count`,
|
||||
`apply`, `=`, vector/map/set constructors, var deref/bind, keyword/symbol
|
||||
construction. The back end maps each to the host (on Janet, mostly the existing
|
||||
`core-*`). To re-host, implement this set.
|
||||
|
||||
## Why name-based vars (not embedded cells)
|
||||
|
||||
`compiler.janet` compiles a global ref to a closure over the Janet var cell. That
|
||||
(a) is a Janet value baked into the IR — not portable, and (b) can't be marshaled
|
||||
for AOT without the runtime-dict trick. Compiling instead to *resolve var by
|
||||
(ns,name) at call time* through an RT primitive keeps redefinition live, makes the
|
||||
IR host-neutral, and makes images trivially portable. The per-call lookup is the
|
||||
cost; it can be cached/direct-linked later as an opt-in optimization.
|
||||
|
||||
## Bootstrap & staging (keeps the suite green throughout)
|
||||
|
||||
`compiler.janet` stays as the **bootstrap back end** until the Clojure pipeline is
|
||||
proven. Order:
|
||||
|
||||
1. **Freeze the IR** spec and refactor `compiler.janet`'s emit to consume
|
||||
name-based `:var` (no behavior change; bootstrap still works).
|
||||
2. **Define the host contract** (`cenv` + `RT`) and implement it on Janet,
|
||||
exposed under a stable namespace the Clojure core can call.
|
||||
3. **Write `jolt.analyzer` in Clojure** producing IR, against `cenv`. Diff its IR
|
||||
against the Janet analyzer on the conformance corpus until identical.
|
||||
4. **Janet back end consumes IR** from the Clojure analyzer; wire into the loader
|
||||
behind a flag. Validate at parity (dual-mode conformance + clojure-test-suite).
|
||||
5. **Flip** the loader to the Clojure analyzer + Janet back end; `compiler.janet`
|
||||
shrinks to the back end only.
|
||||
6. **Move `clojure.core`** macros then fns into `jolt-core` incrementally, each
|
||||
compiled by the prior stage, isolating host bits behind `RT`.
|
||||
|
||||
Guards at every step: the dual-mode conformance harness (interpret vs compile)
|
||||
and the clojure-test-suite baseline.
|
||||
|
||||
## The portability test
|
||||
|
||||
When done, re-hosting Jolt to runtime X means writing only: `host/X/{reader, rt,
|
||||
backend, cenv, bootstrap}`. `jolt-core/{ir, analyzer, macros, core}` is reused
|
||||
unchanged. That is the concrete bar for "truly self-hosted and portable."
|
||||
|
|
@ -1,175 +0,0 @@
|
|||
# Toward a self-hosting Jolt compiler
|
||||
|
||||
Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc
|
||||
compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet
|
||||
bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet
|
||||
bootstrap. This is a design doc, not a changelog — it describes where we are, the
|
||||
prior art, the constraints we verified, and a recommended path.
|
||||
|
||||
## The goal
|
||||
|
||||
- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is
|
||||
enough to start; the rest of Clojure — including the compiler — is written in
|
||||
Clojure and compiled by Jolt itself, growing the language as it compiles more
|
||||
of itself.
|
||||
- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast),
|
||||
not tree-walking.
|
||||
- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols,
|
||||
multimethods, and everything else stay live and redefinable at the REPL even
|
||||
for compiled code.
|
||||
- **Minimal host requirement.** Shrink what must exist in Janet to the
|
||||
irreducible base.
|
||||
|
||||
## Where Jolt is today
|
||||
|
||||
- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a
|
||||
tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the
|
||||
stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in
|
||||
the host, inverted from the Clojure-in-Clojure ideal.
|
||||
- The interpreter (`eval-form`) is the complete reference path.
|
||||
- The compiler (`compiler.janet`) — `analyze-form` (reader form → `:op` AST) →
|
||||
`emit` (AST → Janet form) → Janet `compile`/`eval` — is now **on by default**
|
||||
in the shipped runtime (`JOLT_INTERPRET=1` opts out). It is a *hybrid*: forms
|
||||
it can't compile correctly throw `jolt/uncompilable` and fall back to the
|
||||
interpreter (`loader/eval-toplevel`), so results always match the interpreter.
|
||||
Validated at parity — conformance 218/218 under both interpret and compile, and
|
||||
the clojure-test-suite under compile passes 3932 (vs the 3913 interpreter
|
||||
baseline) across ~4.6k assertions.
|
||||
- Done so far: var-indirection (globals deref through var cells, so compiled code
|
||||
is REPL-redefinable); hybrid fallback; compilation of multi-arity / named /
|
||||
variadic fns and `recur` inside `fn`; map and vector literal compilation
|
||||
(mode-correct via `make-vec` / `build-map-literal`); resolution that mirrors
|
||||
the interpreter (current ns → `clojure.core` → Janet-env fallback); and AOT
|
||||
(`aot.janet`) that marshals a compiled namespace to a Janet bytecode image
|
||||
against the baked-in runtime dictionary and loads it back.
|
||||
- Still open — the actual self-hosting: the compiler and most of `clojure.core`
|
||||
are still Janet. Rewriting them in Clojure (compiled by Jolt) is the remaining
|
||||
Clojure-in-Clojure work.
|
||||
|
||||
## What the host gives us (verified)
|
||||
|
||||
Janet already is the backend and the AOT story — we don't need a custom bytecode
|
||||
emitter:
|
||||
|
||||
- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is
|
||||
Clojure form → correct Janet form → `compile`.
|
||||
- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled
|
||||
environment to a **bytecode image** and load it back: this is Phase 4 AOT.
|
||||
- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the
|
||||
form layer (we shouldn't need to).
|
||||
|
||||
**The catch we verified:** Janet *early-binds* top-level references. Compile
|
||||
`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns
|
||||
the old value. So emitting Jolt globals as plain Janet symbols (what the current
|
||||
compiler largely does) is fundamentally incompatible with REPL redefinition. This
|
||||
is the single most important design constraint below.
|
||||
|
||||
## Prior art
|
||||
|
||||
- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is
|
||||
written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only
|
||||
~20 special forms are primitive; everything else is macros/functions. Crucially,
|
||||
compiled call sites go **through Var objects** (a deref), so redefining a var is
|
||||
visible to existing compiled callers — that's how speed and live redefinition
|
||||
coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the
|
||||
var indirection) for speed where you don't need redefinition (used for core in
|
||||
production). AOT compiles namespaces to `.class` files.
|
||||
- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus
|
||||
a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS).
|
||||
`cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at
|
||||
~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we
|
||||
want with Janet as the backend.
|
||||
- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally
|
||||
specified* intermediate languages, with autogenerated boilerplate to recur
|
||||
through unchanged forms and checks that each pass's output matches its grammar.
|
||||
The lesson for "grow the language as it compiles itself": keep passes small and
|
||||
IRs explicit so adding a form is local and verifiable.
|
||||
- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS
|
||||
(optimization IR) → VM bytecode, with several front-end languages targeting
|
||||
Tree-IL. The closest analog to "Lisp → bytecode on a VM."
|
||||
|
||||
## Assessment: is the current approach the right one?
|
||||
|
||||
The overall *shape* is right and matches ClojureScript: front-end (analyze →
|
||||
emit) with the host VM as the backend, emitting host forms that the host compiles
|
||||
to bytecode. Two things need to change to reach the goal:
|
||||
|
||||
1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref
|
||||
through the var cell**, not as a Janet symbol. Jolt vars are already cells
|
||||
(`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly
|
||||
`((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition
|
||||
updates the cell's root, so compiled callers see it — exactly Clojure's model.
|
||||
One indirection per global call; locals and control flow stay direct and fast.
|
||||
Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition.
|
||||
2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting
|
||||
means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is
|
||||
Clojure. That's the bulk of the work and where the "language builds itself"
|
||||
payoff lives.
|
||||
|
||||
So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for
|
||||
free), fix global binding, and progressively self-host.
|
||||
|
||||
## Recommended architecture
|
||||
|
||||
**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small,
|
||||
named passes rather than one big walker:
|
||||
|
||||
1. *read* — reader → forms (already have it).
|
||||
2. *macroexpand* — fully expand to special forms + calls (the interpreter already
|
||||
expands; share one expander).
|
||||
3. *analyze* — forms → AST, resolving locals vs vars and tagging ops.
|
||||
4. *(optional) optimize* — constant-fold, direct-link hot calls, etc.
|
||||
5. *emit* — AST → Janet form, with globals as var-cell derefs.
|
||||
6. *compile* — Janet `compile` → bytecode; `make-image` for AOT.
|
||||
|
||||
Make each pass total over the IR so an unhandled node is an explicit gap, not a
|
||||
silent miss.
|
||||
|
||||
**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in
|
||||
the host before any Clojure can run: the reader; the value/representation layer
|
||||
(vars, namespaces, symbols, keywords, persistent collections, chars); host
|
||||
interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur`
|
||||
evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`,
|
||||
the macros, and the compiler — is Clojure loaded and (eventually) compiled by the
|
||||
kernel. Today the kernel is far larger than this; shrinking it is a long game.
|
||||
|
||||
**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass
|
||||
can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`)
|
||||
for that sub-form instead of erroring. This lets the compiler be incomplete and
|
||||
still correct (hot paths compile, cold/unsupported paths interpret), lets us grow
|
||||
coverage incrementally, and de-risks the self-hosting bootstrap.
|
||||
|
||||
**Live flexibility.** Vars stay first-class cells; compiled code derefs them;
|
||||
`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct
|
||||
linking seals a call against redefinition, so the interactive modes — the REPL,
|
||||
`-e`, the nREPL server — always stay live (indirect). Running a *program* (a
|
||||
file, `-m`/`-M`) direct-links by default, since it's a closed world; opt back out
|
||||
with `JOLT_NO_DIRECT_LINK`. (See RFC 0005, "Compilation modes and defaults".)
|
||||
|
||||
## A staged path
|
||||
|
||||
1. **Var-indirection in the emitter** — *done*. Global refs compile as var-cell
|
||||
derefs, so a compiled `defn` is redefinable at the REPL.
|
||||
2. **Hybrid fallback + coverage** (`jolt-1bj`) — *done*. Forms the compiler can't
|
||||
compile throw `jolt/uncompilable` and fall back to the interpreter, so compile
|
||||
mode is always correct. Covered: multi-arity/named/variadic fns, `recur` in
|
||||
`fn`, map/vector literals, and resolution matching the interpreter.
|
||||
Destructuring compiles via the shared `destructure` expander: the `fn`/`let`/
|
||||
`loop`/`defn` macros desugar to plain-symbol `fn*`/`let*`/`loop*`, so it no
|
||||
longer falls back — and the primitives reject patterns outright, matching
|
||||
Clojure (`jolt-f79`).
|
||||
5. **Compile-by-default + AOT** (`jolt-7j9`) — *done, done out of order*. Once the
|
||||
hybrid path was validated at parity, compilation was flipped on by default and
|
||||
AOT images (`aot.janet`) landed. Done before 3–4 because it's the runtime
|
||||
payoff and only needed the hybrid path to be correct, not self-hosting.
|
||||
3. **Self-host the compiler** (`jolt-lcn`) — *open*. Rewrite `compiler.janet` as
|
||||
Clojure (`jolt.compiler`) that Jolt compiles. Now the compiler is part of the
|
||||
language it compiles.
|
||||
4. **Shrink the kernel / core-in-Clojure** (`jolt-uqi`) — *open*. Move
|
||||
`clojure.core` from Janet to Clojure incrementally, each piece compiled by the
|
||||
previous stage — the language building itself — leaving a minimal Janet kernel.
|
||||
|
||||
What remains (3 and 4) is the actual Clojure-in-Clojure rewrite: the largest part
|
||||
of the work and where the "language builds itself" payoff lives. The correctness
|
||||
and runtime foundations it needs — redefinable compiled code, an always-correct
|
||||
hybrid path, compile-by-default, and AOT — are now in place.
|
||||
|
|
@ -68,8 +68,9 @@ Behavioral questions are settled in this order: differential testing against
|
|||
the reference implementation → cross-dialect agreement in clojure-test-suite
|
||||
→ ClojureDocs community examples (verified before inclusion) → reference
|
||||
source (for intent). Conformance tests live in this repository
|
||||
(`test/integration/conformance-test.janet` runs each assertion through three
|
||||
independent execution paths) and in the cross-dialect clojure-test-suite.
|
||||
(the corpus `test/chez/corpus.edn`, run on Chez via `host/chez/run-corpus.ss`
|
||||
and certified against reference JVM Clojure by `test/conformance/certify.clj`)
|
||||
and in the cross-dialect clojure-test-suite.
|
||||
|
||||
## 5. Chapter plan
|
||||
|
||||
|
|
|
|||
|
|
@ -38,10 +38,13 @@ Of the 694 `clojure.core` vars in the ClojureDocs inventory:
|
|||
|
||||
## How this connects to the test suites
|
||||
|
||||
- `test/integration/conformance-test.janet` — 302 assertions, each run
|
||||
through three independent execution paths (interpreter, bootstrap
|
||||
compiler, self-hosted compiler) that must agree. Spec entries cite these.
|
||||
- `test/spec/*.janet` — ~1,500 behavioral cases organized by topic.
|
||||
- `test/chez/corpus.edn` — the host-neutral behavioral corpus, one row per
|
||||
case (`{:suite :label :expected :actual}`). The Chez compiler evaluates each
|
||||
case via `host/chez/run-corpus.ss` (run with `make corpus`), and
|
||||
`test/conformance/certify.clj` certifies every `:expected` against reference
|
||||
JVM Clojure (run with `make certify`). Spec entries cite these cases.
|
||||
- `test/conformance/` — the certification tooling and classified divergences
|
||||
(`certify.clj`, `known-divergences.edn`); see its `README.md` and `SPEC.md`.
|
||||
- `vendor/clojure-test-suite` — the cross-dialect suite (≥4081 assertions
|
||||
passing); dialect splits there are classification evidence.
|
||||
- jank's per-construct corpus (`~/src/jank/compiler+runtime/test/jank`) is
|
||||
|
|
|
|||
|
|
@ -9,91 +9,57 @@ Scope, decided up front:
|
|||
- **pure `clj`/`cljc`** — anything needing the JVM won't load or run; expected.
|
||||
- **no classpath abstraction** — `require` just needs to find a dep's namespaces;
|
||||
"the classpath" is an ordered list of source directories.
|
||||
- **piggyback on jpm** — reuse jpm's git fetch + cache; don't write a package
|
||||
manager.
|
||||
- **own resolver, own reader** — `deps.edn` is read by jolt's own reader, and git
|
||||
fetch/cache is a thin shell-out to `git`; no external package manager.
|
||||
- **deps-agnostic runtime core** — resolution is a CLI front-end concern, not a
|
||||
runtime one. The `jolt` *runtime* knows nothing about deps.edn; it only reads
|
||||
source roots from `JOLT_PATH`. The `jolt` *CLI* resolves a deps.edn into that
|
||||
env var before running, in a module (`deps.janet`) that loads `jpm` lazily.
|
||||
(This was a separate `jolt-deps` binary originally; it was folded into `jolt`
|
||||
for a single-binary UX — the code boundary stayed, only the executable merged.
|
||||
A back-compat `jolt-deps` shim still ships and forwards to `jolt`.)
|
||||
runtime one. The runtime knows nothing about `deps.edn`; it only consumes a
|
||||
list of source roots. The CLI resolves a `deps.edn` into those roots before
|
||||
running.
|
||||
|
||||
## How jpm handles dependencies
|
||||
## How resolution works
|
||||
|
||||
jpm's package code (`jpm/pm.janet`) splits into a fetch half and a build half,
|
||||
and we use only the first:
|
||||
`jolt.deps` (`jolt-core/jolt/deps.clj`) reads `deps.edn` (jolt's own reader
|
||||
parses the EDN), then walks `:deps`:
|
||||
|
||||
- **`resolve-bundle`** normalizes a dep spec to `{:url :tag :type :shallow}`,
|
||||
accepting `:url`/`:repo` + `:tag`/`:sha`/`:commit`/`:ref`. A deps.edn
|
||||
`{:git/url … :git/sha …}` maps straight onto it.
|
||||
- **`download-bundle url :git tag shallow`** clones into a content-addressed cache
|
||||
(`<modpath>/.cache/git_<tag>_<sanitized-url>`) and returns the path —
|
||||
`git init` + `remote add` + fetch + reset, plus submodules. No build step.
|
||||
- **`bundle-install`** is the half we skip: it then runs `project.janet` build
|
||||
rules, which a Clojure lib doesn't have. It's cleanly separable from the clone.
|
||||
|
||||
So jpm gives us git resolution and a cache for free; calling `download-bundle`
|
||||
needs `jpm/config/load-default` first (it sets `gitpath` and the cache dyns).
|
||||
|
||||
## How it works
|
||||
|
||||
`src/jolt/deps.janet` reads `deps.edn` (Janet parses it directly — EDN and Janet
|
||||
syntax overlap for the `:deps`/`:paths` subset), then walks `:deps`:
|
||||
|
||||
- `:git/url` (+ `:git/sha` or `:git/tag`) → `resolve-bundle` + `download-bundle`
|
||||
into `jpm_tree/.cache`;
|
||||
- `:git/url` + `:git/sha` (+ optional `:deps/root`) → clone the sha into the git
|
||||
cache and contribute the checkout (or its `:deps/root` subdir);
|
||||
- `:local/root` → the path as-is;
|
||||
- `:mvn/*` and anything else → ignored.
|
||||
- `:mvn/*` → skipped with a warning;
|
||||
- anything else → ignored.
|
||||
|
||||
git resolution shells out to `git` through `jolt.host/sh` — `git init` + remote
|
||||
add + fetch + reset at the requested sha. Clones land in a global, sha-immutable
|
||||
cache (`$JOLT_GITLIBS`, else `~/.jolt/gitlibs`) shared across projects, the
|
||||
`tools.gitlibs` `~/.gitlibs` model.
|
||||
|
||||
Each resolved dependency contributes its own `:paths` (default `["src"]`) as
|
||||
source roots; the walk is **breadth-first** so every top-level coordinate
|
||||
registers before any transitive one — a top-level pin always wins, matching
|
||||
tools.deps, and a coordinate conflict warns on stderr naming both. The result
|
||||
is a de-duplicated, ordered list of directories. `resolve-deps-cached` memoizes
|
||||
that list in the project-local `.cpcache/jolt-deps.jdn`, keyed on a hash of the
|
||||
project `deps.edn` + the user-level `deps.edn` + the selected aliases. jpm is
|
||||
loaded lazily (`require`, not `import`) so it's pulled in only when resolving —
|
||||
never embedded in a built binary.
|
||||
tools.deps. The result is a de-duplicated, ordered list of directories.
|
||||
|
||||
Three tools.deps features are mirrored in reduced form. **Aliases**: `:aliases`
|
||||
Two tools.deps features are mirrored in reduced form. **Aliases**: `:aliases`
|
||||
entries supply `:extra-paths`/`:extra-deps` (accumulate across the aliases
|
||||
selected with `-A:a:b`) and `:main-opts` (last-wins, run with `-M:alias`).
|
||||
**User config**: a `deps.edn` under `$JOLT_CONFIG` (else
|
||||
`$XDG_CONFIG_HOME/jolt`, else `~/.jolt`) merges beneath the project file,
|
||||
per key, project wins. **Tasks**: the honest subset of babashka's — a string
|
||||
task is a shell command, a map task is `{:main-opts […] :doc "…"}`; bare
|
||||
Clojure expressions aren't supported because the reader hands back parsed
|
||||
data, and round-tripping it to source isn't worth the fragility.
|
||||
**Tasks**: the honest subset of babashka's — a string task is a shell command, a
|
||||
map task is `{:main-opts […]}`; bare Clojure expressions aren't a separate task
|
||||
form.
|
||||
|
||||
Clones default to a global sha-immutable cache (`$JOLT_GITLIBS`, else
|
||||
`<config-dir>/gitlibs`) shared across projects, the `tools.gitlibs`
|
||||
`~/.gitlibs` model; per-project trees remain available by passing `tree`
|
||||
explicitly.
|
||||
## How the CLI ties it together
|
||||
|
||||
The loader (`evaluator.janet/find-ns-file`) resolves a namespace by searching the
|
||||
context's `:source-paths` in order (the stdlib `src/jolt` first), trying `<ns>.clj`
|
||||
then `<ns>.cljc`. Extra roots come from `JOLT_PATH` or `init`'s `:paths` option.
|
||||
`jolt.main` (`jolt-core/jolt/main.clj`) is the CLI dispatch. Driven by `cli.ss`,
|
||||
it resolves the project (`jolt.deps/resolve-project`), prepends the resolved
|
||||
roots, and de-sugars the argv into a run:
|
||||
|
||||
The `jolt` CLI (`src/jolt/main.janet`, `resolve-deps-argv`) ties it together: on
|
||||
a deps subcommand — or any runnable command in a directory that has a `deps.edn`
|
||||
— it resolves the roots, sets `JOLT_PATH`/`JOLT_APP_PATHS`, and de-sugars the
|
||||
argv into a plain runtime command (`-M:alias` → the alias `:main-opts`, `run
|
||||
FILE` → `FILE`, …) that the normal dispatch then runs. `main.janet` imports
|
||||
`deps.janet`, so the resolver ships in the `jolt` binary; but `deps.janet` loads
|
||||
`jpm` lazily, and the runtime modules (`api`/`backend`/RT) never import it, so an
|
||||
app baked from its own `jolt/api` entry doesn't link it. The runtime's only
|
||||
dependency interface remains that one env var.
|
||||
- `run -m NS args` → load `NS`, call its `-main`;
|
||||
- `run FILE` → load the file;
|
||||
- `-M:alias` → run the alias's `:main-opts`;
|
||||
- `-A:alias` → add the alias's paths/deps, then run the rest;
|
||||
- `repl` → a line REPL;
|
||||
- `path` → print the resolved roots;
|
||||
- `<task>` → run a `deps.edn` `:tasks` entry.
|
||||
|
||||
`jolt uberscript` bundles a namespace and everything it requires into one
|
||||
standalone `.clj`. It requires the entry namespace and uses the order in which
|
||||
the loader finishes loading files — a dependency finishes before the file that
|
||||
required it, so the order is topological — then concatenates that source. The
|
||||
baked-in stdlib is excluded (it's part of the runtime, not bundled).
|
||||
|
||||
Gotcha worth remembering: the `jolt` CLI's context is built into its image at
|
||||
build time, so `JOLT_PATH` is applied at runtime in `main`, not in `init` (whose
|
||||
env read would be frozen at build).
|
||||
The resolver lives in the overlay alongside the runtime, but the runtime's only
|
||||
dependency interface is the list of source roots it's handed.
|
||||
|
||||
## Limitations
|
||||
|
||||
|
|
@ -105,35 +71,9 @@ env read would be frozen at build).
|
|||
|
||||
## Conformance
|
||||
|
||||
`test/integration/deps-conformance-test.janet` resolves a few real pure-`cljc`
|
||||
git libraries and reports whether their namespaces load and a sample call works.
|
||||
It's network-gated behind `JOLT_CONFORMANCE=1` so CI stays offline. Use it to
|
||||
check a library against the current interpreter, and to drive fixes for whatever
|
||||
gap a failure points at (the same loop as the clojure-test-suite battery). A
|
||||
library fails when it relies on something Jolt doesn't provide — JVM interop, or
|
||||
a regex feature like Unicode property classes (`\p{…}`).
|
||||
|
||||
## Not yet
|
||||
|
||||
- **Compiling deps into a binary image.** `uberscript` already produces a
|
||||
standalone `.clj`; baking a project's dependencies directly into a custom
|
||||
executable image is a heavier variant that isn't implemented.
|
||||
|
||||
## Janet dependencies: `:jpm/module`
|
||||
|
||||
A jolt project can depend on janet libraries. jpm owns their installation;
|
||||
`deps.edn` declares the requirement and `jolt` verifies it at resolve time:
|
||||
|
||||
```clojure
|
||||
:deps {janet/spork-http {:jpm/module "spork/http"
|
||||
:jpm/install "spork"}}
|
||||
```
|
||||
|
||||
- `:jpm/module` — the janet module path that must be importable.
|
||||
- `:jpm/install` (optional) — the jpm package to install when it isn't;
|
||||
`jolt` runs `jpm install <name>` once, then re-checks. Without it the resolve
|
||||
fails with the install hint.
|
||||
|
||||
A `:jpm/module` dep contributes no source roots. At runtime the `janet.*`
|
||||
interop bridge autoloads the module on first reference
|
||||
(`janet.spork.http/server`, …), so nothing else is needed.
|
||||
The known-working libraries (see [libraries.md](libraries.md)) and the
|
||||
[examples](https://github.com/jolt-lang/examples) exercise real pure-`cljc` git
|
||||
libraries end to end — resolving them from git, loading their namespaces, and
|
||||
running sample calls. A library fails when it relies on something Jolt doesn't
|
||||
provide — JVM interop, or a regex feature like Unicode property classes
|
||||
(`\p{…}`).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue