diff --git a/CLAUDE.md b/CLAUDE.md index 5f05eba..89d3e3b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -54,7 +54,7 @@ bd close # Complete work ## Build & Test No build step — `bin/joltc` runs off the checked-in seed (`host/chez/seed/`). -The gate is pure Chez (+ Clojure for the JVM oracle); no Janet. +The gate is pure Chez (+ Clojure for the JVM oracle). ```bash bin/joltc -e EXPR # run a Clojure expression on Chez @@ -79,7 +79,7 @@ selfhost` fails. Runtime-only `host/chez/*.ss` shims do NOT need a re-mint. ## Architecture Overview -Clojure on Chez Scheme — the sole substrate, no Janet. A small Chez runtime +Clojure on Chez Scheme — the sole substrate. A small Chez runtime (`host/chez/*.ss`: value model, persistent collections, seqs, vars/ns, host interop) hosts a portable Clojure overlay (`jolt-core/`): the reader/analyzer/IR/backend (`jolt-core/jolt/`) and `clojure.core` in diff --git a/README.md b/README.md index a6a8943..5956a3f 100644 --- a/README.md +++ b/README.md @@ -77,10 +77,9 @@ Jolt targets Clojure semantics but runs on Chez, not the JVM. and `go`/`!`/`alts!`/`timeout`. - **Regex.** Backed by [irregex](https://github.com/ashinn/irregex) (vendored), PCRE/Java-style patterns. -- **Collections.** Immutable persistent vectors (32-way tries), cons lists, and HAMT - maps/sets. Hash-map/hash-set iteration order is unspecified — use - `sorted-map`/`sorted-set` when order matters. Transients are real mutable scratch - collections. +- **Collections.** Immutable persistent vectors, cons lists, and HAMT maps/sets. + Hash-map/hash-set iteration order is unspecified — use `sorted-map`/`sorted-set` + when order matters. Transients are real mutable scratch collections. Supported and Clojure-compatible: lazy/infinite sequences, transducers, destructuring, multimethods with hierarchies, protocols/records @@ -91,12 +90,14 @@ destructuring, multimethods with hierarchies, protocols/records ## Test ```bash -make test # the full gate (no Janet) +make test # the full gate make corpus # conformance corpus vs the JVM-sourced spec make unit # host-specific unit cases make selfhost # bootstrap fixpoint (rebuild == checked-in seed) make smoke # bin/joltc CLI smoke make sci # load borkdude/sci's source through joltc (compat stress) +make ffi # HTTP-server GC-safety + http-client temp paths +make transient # transient mutation + linear-time builds make certify # JVM oracle (skips if clojure is absent) ``` diff --git a/bench/README.md b/bench/README.md index 01ab8bc..549a5c1 100644 --- a/bench/README.md +++ b/bench/README.md @@ -53,7 +53,6 @@ the broadening (2026-06-16), ratios cluster by axis: ## Running ```sh -jpm build && export PATH="$PWD/build:$PATH" bench/run.sh # whole-program optimization on (default) JOLT_WHOLE_PROGRAM=0 bench/run.sh # WP off, to measure what WP buys bench/run.sh binary-trees 16 # one benchmark, custom size @@ -62,7 +61,7 @@ bench/run.sh binary-trees 16 # one benchmark, custom size ## A/B against a change To measure a pass, run the suite on `main`, then on the branch, back to back -(same machine, quiet) — the protocol used for `test/bench/core-bench.janet` and -the ray tracer. Each benchmark prints `runs: [...]` and `mean: N ms`; compare +(same machine, quiet) — the same protocol used for the ray tracer. Each +benchmark prints `runs: [...]` and `mean: N ms`; compare the means. A pass is worth landing when it moves a benchmark whose axis it targets, even if the ray tracer stays flat. diff --git a/docs/architecture-refactor-plan.md b/docs/architecture-refactor-plan.md deleted file mode 100644 index 7be4b73..0000000 --- a/docs/architecture-refactor-plan.md +++ /dev/null @@ -1,267 +0,0 @@ -# Architecture Refactor Plan - -Goal: make the jolt codebase easier to understand and safe to change — for human -maintainers and, specifically, for LLM agents. An agent should be able to find -where a feature lives, see its related code, and make a change without scanning -3000-line files or keeping invisible global state in mind. - -This plan synthesizes a six-part architectural review (one reviewer per -subsystem). It is organized as **independent, gate-validated phases**, ordered by -value-to-risk. Each phase is a PR-sized unit. Nothing here changes behavior; it is -pure reorganization plus a small set of dead-code/bug deletions. - -## Non-negotiable constraints (apply to every phase) - -- **Every phase passes the full gate**: `jpm test` green (conformance ×3 modes, - `clojure-test-suite` ≥ baseline, bench back-to-back vs main, real exit code). - `rm -rf build && jpm clean` before trusting the binary. -- **Load order is load-bearing.** The Janet seed (`src/jolt/*.janet`) and the - Clojure overlay tiers (`jolt-core/clojure/core/NN-*.clj`) load in a fixed order; - a module may only use what loads before it. Splitting a file must preserve the - net load order (a new file is imported where the old code ran). -- **Seed-tier discipline** (the jolt-tzo rules in CLAUDE.md): nothing the - analyzer/IR use may move below the kernel tier; a tier may only use macros from - earlier tiers; expander-called fns stay in 00-syntax. -- **One concern per PR.** Do not combine a file split with a behavior fix. - Dead-code/bug deletions (Phase 0) land first and separately so later diffs are - pure moves. - -## Guiding principles (the target state) - -1. **One obvious home per feature.** Adding a `.someMethod` interop shim, a reader - macro, or a clojure.core fn should have exactly one file an agent edits. -2. **Files map to concerns, not to history.** No 3000-line grab-bags; no module - whose name lies about its contents (`javatime.janet` is 80% not java.time; - `phm.janet` contains LazySeq; `types.janet` holds seven concerns). -3. **Make implicit contracts explicit and checked.** The seed↔overlay split, the - ctx-shaping env-knob list, and the IR op set are tribal knowledge today; turn - each into a single source of truth with a drift check. -4. **No copy-paste dispatch.** Where the same op-set / member-dispatch / cache - dance is hand-written N times, extract one combinator. - ---- - -## Phase 0 — Dead code & concrete bugs (low risk, do first) - -Pure deletions and small fixes, each with a regression row. These remove -*actively misleading* code (comments that contradict behavior) before any moves. - -| Item | Location | Note | -|---|---|---| -| `find` defined twice; the dead copy returns a plain vector and is *wrong* | `jolt-core/clojure/core/20-coll.clj:347-349` | live def at :787 returns a real map-entry; delete the dead one | -| `core-satisfies?` always returns `false` (latent bug + misleading comment) | `src/jolt/core.janet:2412` | either implement over the protocol registry or document why inert; fix the `eval-list` comment that claims it's an overlay fn | -| `File.toURL` stores `:url` but every `:jolt/url` method reads `:spec` | `src/jolt/javatime.janet:636` | broken shim; use `:spec`, add a spec row | -| `core-type->str` — zero references | `src/jolt/core.janet:2416` | delete defn + binding | -| `core-resolve` — unreachable (overridden by `install-stateful-fns!`) | `src/jolt/core.janet:2365` | delete defn + binding; fix stale comment | -| `mark-hint` — unreachable | `jolt-core/jolt/passes.clj:836` | delete | -| `pad2` defined twice | `src/jolt/javatime.janet:41,522` | keep one | -| `phs-to-struct`, `shape-vals`, `ns-imports-fn` — zero references | `phm.janet:302`, `types.janet:655`, `types.janet:527` | delete unless reserved API | -| `pl-rest` no-op `(if (plist? r) r r)` | `src/jolt/plist.janet:62` | collapse / fix; regression check | -| `read-quote` unused `pos` param | `src/jolt/reader.janet:608` | drop | -| Stale/contradictory comments (`extend`, orphan `;; trampoline:` / `;; rand-int:` headers, migration breadcrumbs) | `30-macros.clj:402-404`, `20-coll.clj:558-560,780,790` | sweep | -| **`:map-shapes?` missing from the deps-image cache key** (possible stale-image correctness gap) | `src/jolt/main.janet:440-448` | add to key; confirm with a test | - ---- - -## Phase 1 — Extract the host-interop subsystem (highest value) - -**Problem (corroborated by 3 reviewers).** The JVM-emulation shim layer is the -single worst sprawl, and it is exactly where the recent hiccup/markdown/malli work -landed ad hoc. A single class is split across up to three files: - -- registry *machinery* (`class-statics`/`tagged-methods`/`class-ctors` + - `register-*!`) lives in `evaluator.janet:624-651`; -- four static tables (`Math`/`Thread`/`System`/`Long`/`Number`) are hardcoded in - `evaluator.janet:537-619` and dispatched *outside* the registry in `resolve-sym` - (780-794); instance-method tables (`string/number/object-methods`, 698-775) are - likewise inline; -- the bulk of the shims are in `javatime.janet` (674 lines, ~80% not java.time: - java.io/util/net/nio/sql/text/math all in one `install-io!`); -- `api.janet:18-56` wires the malli statics + `set-coll-interop!` as load-time - side effects in the public-API module; -- `core.janet` holds `File`/JDBC constructors; `types.janet` holds the `:jolt/inst` - representation. - -**Target.** A `src/jolt/interop/` directory, one mechanism file + one file per JDK -area, each owning a class's *full* surface (ctor + statics + methods + `instance?` -predicate + canonical name) and exposing one `install!`: - -``` -src/jolt/interop/ - registry.janet # MOVED from evaluator: class-statics/tagged-methods/class-ctors, - # register-*!, canonical-names, value-overrides, instance? registry - coerce.janet # shared: chr, char->byte, render-piece/writer-piece, pad2, - # the date-format token walker (kills the pad2/render-piece dups) - java_time.janet java_io.janet java_net.janet java_util.janet - java_lang.janet # absorbs evaluator's inline Math/Thread/System/Long/Number + - # string/number/object method tables — the registry becomes the - # ONLY static/method-dispatch mechanism - jdbc.janet - install.janet # (install-all! ctx) — the ONE place api.janet calls (replaces api 18-56) -``` - -`evaluator.janet`'s `resolve-sym` and the dot-dispatch consult only the registry. -`regex.janet` and `async.janet` stay put — they are engines/library-ports consumed -by interop, not shims. - -**Cheap 80% if the full split is too big for one PR:** rename -`javatime.janet`→`host_interop.janet`, pull the four inline static tables + three -method tables out of `evaluator.janet` into it, move `api.janet:18-56` into it, -dedup `pad2`/`chr`/`render-piece`. Collapses the scatter from 5 files to 2. - -**Risk:** medium — touches the hot dot-dispatch path; load order must keep the -registry available before any `install!`. Validate with `host-interop-spec` + -the hiccup/markdown/malli example apps + full gate. - ---- - -## Phase 2 — Decompose the god-files - -The three biggest interpreter/runtime files are the top LLM-navigability tax. -Split along the cohesive clusters the review mapped (line ranges in the review -notes). Each split is a mechanical move + import; behavior unchanged. - -### 2a. `evaluator.janet` (2681 → ~5 files) -- `special_forms.janet` — explode the **680-line `eval-list`** (1921-2599) into - named `eval-
` fns (`eval-fn*`, `eval-let*`, `eval-try`, `eval-dot`, …) + - a dispatch table. This is the highest-leverage single change: today "where is - `try` handled" means scanning a 680-line body. -- `resolution.janet` — `resolve-sym`/`resolve-var`/binding/destructuring. -- `ns_loader.janet` — module/bridge plumbing + require/in-ns/use/import/refer. -- `runtime_registration.janet` — protocol/defmulti/deftype/reify setup + - `install-stateful-fns!` (1506-1909) split into per-domain registration fns. -- (host-interop already left in Phase 1.) -- **De-dup the two `.method` dot arms** (2456-2507 vs 2512-2550): extract one - `dispatch-member [target field-name args]` used by both. They are copy-pasted - today and must be hand-synced on every interop change. - -### 2b. `core.janet` (3017 → ~6 leaf files) -Clusters are already sequential and mostly leaf — low risk: -`core-types` (predicates/eq/arith/bits), `core-coll` (assoc/seq/transducers/lazy/ -transients), `core-print` (pr-str/str), `core-io`, `core-refs` (atoms/vars/delays/ -arrays/type), `core-bindings` (the table + `init-core!` + a *labeled* stub -section). The `core-bindings` table stays the single registration point. - -### 2c. `passes.clj` (1487 → 3-4 files behind one façade) -The Louvain communities and the cycle analysis agree: the IR-rewriting passes and -the type subsystem are weakly coupled (only `run-passes` + `dirty` shared). -- `jolt/passes.clj` (keep, ~150 lines): `run-passes` + shared state + re-exports — - the only file the back end imports. -- `jolt/passes/fold.clj` — const-fold (always-on). -- `jolt/passes/inline.clj` — inline + flatten-lets + scalar-replace (share the - alpha-rename invariant). -- `jolt/passes/types.clj` — type lattice + `infer` + success-checker + driver API - (kept as one module to respect the inference↔checker cycle; no `declare` - gymnastics). Extract the `infer` `:invoke` arm (1051-1160) into per-shape - helpers regardless of the split. -- Fix the **stale ns docstring** (lists 4 passes, omits the type system that is - >50% of the file). - ---- - -## Phase 3 — Kill the structural duplication - -### 3a. One IR op-walk combinator -There are **11 hand-written recursive walks** over the IR op set (const-fold, -inline, subst, body-closed?, pure?, flatten-lets, local-escapes?, subst-lookup, -scalar-replace, infer, backend `emit`). Adding an IR op means editing all of them, -and the "unknown ops pass through" promise is only partly kept. Introduce one -`map-ir-children` (in `ir.clj` or `jolt/ir/walk.clj`) that knows each op's child -positions; rewrite the walks as `(map-ir-children f node)` + their few specials. -Collapses ~400 lines and makes adding an op a one-site change. - -### 3b. IR shape hygiene -`:let`/`:loop` bindings are `[name init]` vectors, `:map` pairs are `[k v]`, `:fn` -arities are maps; optional keys are present-or-absent, which is *why* the backend -needs `norm-node`/phm-densification everywhere. Make `ir.clj` constructors always -emit optional keys (nil-valued); delete the defensive `norm-node` calls. - -### 3c. Smaller dedups -- `read-delimited` driver for the 4 collection readers (`read-list/vec/set/kvs`) - in `reader.janet` (the skip/splice logic already drifted once). -- `bucket-index-of` for the 5 stride-2 bucket scans in `phm.janet`. -- Unify the kw-lookup head-matching reimplemented in analyzer/passes/backend - (a kw-lookup the inference tags but the backend doesn't specialize is a silent - miss) behind one shared predicate + fn-name table. - ---- - -## Phase 4 — Config & caching coherence - -### 4a. Lift run-mode/config resolution into `config.janet` -`config.janet` is 15 lines (one constant); meanwhile `main.janet:585-630` holds -~45 lines of pure env-knob policy (`open-mode?`/`dl`/`optimize?`/shape/whole-program -gates) that can't be unit-tested without the CLI and that the cache keys need to -share. Promote it: `config/resolve-run-mode [argv env] → {:direct-linking? :inline? -:shapes? :map-shapes? :whole-program? :direct-link-auto?}`, plus a canonical -`ctx-shaping-env-vars` list. `main` shrinks to: parse argv → resolve → install → -dispatch. - -### 4b. One ctx-image module, two policies -`init-cached` (core image, api.janet) and the deps-image (main.janet) duplicate the -fork→validate→reinstall-print-cb→atomic-publish dance, each with a **hand-built -positional `%q|...` cache key** that silently misaligns if a knob is added. -Extract `ctx_image.janet`: `load-image [path predicate]`, `save-image [ctx path]`, -`ctx-cache-key [pairs]` (derived from `ctx-shaping-env-vars`, impossible to -misalign). Both callers differ only in the validity predicate (source-fingerprint -vs mtime-manifest). This also closes the cache-key footgun for good and resolves -`aot.janet`'s status (fold its marshal helpers in, or delete it — it is exercised -only by one integration test and is on no run path). - ---- - -## Phase 5 — Data structures, reader, types, and the seed↔overlay index - -### 5a. `types.janet` (699 → 3-4 files) -Seven concerns share a generic name. Split: `value.janet` (char/inst/uuid), -keep the cohesive `var`+`ns`+`ctx` core, `protocols.janet` (the protocol/type -registry, `type-satisfies?`), `records.janet` (shape-records). Minimum win: -extract the protocol registry + shape-records (the two least "types"-like). - -### 5b. `phm.janet` (303 → 3 files) -Split out `lazyseq.janet` (LazySeq has nothing to do with hash maps — pure -mislabel) and `phs.janet` (PersistentHashSet). `phm.janet` keeps the map. - -### 5c. Internal collection protocol -phm/pv/plist each re-implement count/seq/conj/predicate/meta with diverging -naming, and `core.janet` dispatches on them with giant per-op `cond`s (every new -structure edits every cond). A minimal `:jolt/type`-keyed vtable (`-count -seq --conj`) lets core dispatch once. Normalize the trio's naming (`pv` vs `pvec`, -`->`/`-to-`, `EMPTY` vs `EMPTY-PLIST`). - -### 5d. Make the seed↔overlay boundary self-documenting -Five fns exist in both the seed (`core-X`) and overlay (`X`) as dispatch twins with -no cross-reference; nothing indexes which copy is authoritative (`transduce` is -overlay-public but `into` is seed-public — surprising and undocumented). Add: -- a generated `REGISTRY` (name → home → public-source → seed-twin? → dispatch-only?) - with a build-time drift check; -- `SEED-TWIN:` provenance comments on each twin (greppable); -- a distinct prefix for dispatch-only seed helpers so they don't read like public - ones. Mirrors the existing "delete the seed defn + binding in the same change" - rule. - -### 5e. Boundary doc-comments -Add rep-vs-API pointers between the data structures and `core.janet` (e.g. "the -persistent vector trie lives in `pv.janet`; Clojure-facing vector ops and -tuple/pvec polymorphism live in `core.janet`"), so an agent grepping "vector" in -core knows where the representation is. - ---- - -## Sequencing & rationale - -``` -Phase 0 (dead code/bugs) — independent, do first, unblocks clean diffs -Phase 1 (host-interop extract) — highest value; isolates the recent shim sprawl -Phase 2 (god-file splits) — biggest navigability win; 2a/2b/2c independent -Phase 3 (op-walk + IR hygiene) — removes the largest single duplication tax -Phase 4 (config + caching) — fixes the cache-key footgun; makes boot legible -Phase 5 (data/reader/types + index) — finishes the "one home per feature" goal -``` - -Phases are independent; within a phase the sub-items (2a/2b/2c, etc.) are separate -PRs. Highest LLM-friendliness per unit risk: **Phase 1**, **2a (`eval-list` split)**, -**3a (op-walk combinator)**, and **5d (seed↔overlay index)**. - -Each PR: one concern, full gate green, no behavior change (Phase 0 deletions carry -a regression row). diff --git a/docs/building-and-deps.md b/docs/building-and-deps.md index 4bece89..7251486 100644 --- a/docs/building-and-deps.md +++ b/docs/building-and-deps.md @@ -1,40 +1,32 @@ # Building and dependencies -How to build Jolt from source and how to pull Clojure libraries into a project. +How to run Jolt from source and how to pull Clojure libraries into a project. -## Building +## Running ```bash git clone https://github.com/jolt-lang/jolt.git cd jolt git submodule update --init # vendor/sci (used by the SCI bootstrap tests) -jpm build +bin/joltc -e '(println "hello")' ``` -This produces `build/jolt` — one binary that is both the runtime (REPL, -file/expr runner, nREPL server) and the dependency front-end (`deps.edn` -resolution, see below). The whole `.clj` standard library -(`clojure.string`/`set`/`walk`/`edn`/`zip`, `jolt.http`/`interop`/`shell`/ -`nrepl`) is baked in at build time, so it loads from any directory — the artifact -is self-contained. (`clojure.core` is built into the runtime in Janet and -auto-referred, so it's always available.) +There is **no build step**. `bin/joltc` (`host/chez/cli.ss`) loads the +checked-in bootstrap seed (`host/chez/seed/{prelude,image}.ss`) plus the spine +and compiles+evals on Chez (read → analyze → IR → emit → eval), so a fresh +clone runs immediately. The whole `.clj` standard library +(`clojure.string`/`set`/`walk`/`edn`/`pprint`/…) and `clojure.core` are part of +the overlay, so they're always available. -The runtime **core** stays deps-agnostic: it only reads source roots from -`JOLT_PATH`. Dependency resolution lives in a separate CLI front-end module -(`src/jolt/deps.janet`) that the `jolt` entry point calls *before* running your -code, and that lazily loads `jpm` (for git fetch + cache) only when it actually -resolves. So a run with no `deps.edn` never touches the resolver, and an app -baked from its own entry — which imports `jolt/api`, not the CLI — never links -it at all. (`build/` also contains a `jolt-deps` shim that just forwards to -`jolt` so old scripts keep working; prefer calling `jolt` directly.) +`bin/joltc` is both the runtime (REPL, file/expr runner) and the dependency +front-end (`deps.edn` resolution, see below). A run with no `deps.edn` never +touches the resolver. -Needs `jpm` and a recent Janet — developed and CI-tested against **1.41**. The -futures and core.async layers use Janet's threaded `ev/` channels (`ev/thread`, -`ev/thread-chan`), so older Janets may not run the full suite. - -`jpm build` doesn't always notice source changes; run `jpm clean && jpm build` -after editing `src/` to be sure the binaries are current. `jpm test` runs against -the source directly, so it never goes stale. +The bootstrap seed is **checked in**. After changing a seed source — the reader +(`host/chez/reader.ss`), the analyzer/IR/backend (`jolt-core/jolt/*.clj`), or the +`clojure.core` overlay (`jolt-core/clojure/core/*.clj`) — re-mint the seed with +`make remint` (it iterates `host/chez/bootstrap.ss` to a byte-fixpoint), or +`make selfhost` fails. Runtime-only `host/chez/*.ss` shims don't need a re-mint. ## How namespaces are found @@ -47,99 +39,74 @@ come from: at runtime; - the `:paths` option to `init` when embedding Jolt as a library. -If a namespace isn't found on any root, the loader falls back to the stdlib baked -into the binary — that's how `clojure.string` and friends resolve when you run -the binary outside the source tree. +If a namespace isn't found on any root, the loader falls back to the stdlib in +the overlay — that's how `clojure.string` and friends resolve when you run +outside the source tree. So you can point Jolt at a directory of Clojure source with no deps machinery at all: ```bash -JOLT_PATH=/path/to/lib/src build/jolt myfile.clj +JOLT_PATH=/path/to/lib/src bin/joltc run myfile.clj ``` ## Dependencies via deps.edn -`jolt` reads a `deps.edn` in the current directory, fetches its dependencies, -and puts the resolved source directories on `JOLT_PATH` for the run. A `deps.edn` -in the working dir is **auto-resolved** for the runnable commands (`repl`, `-m`, -`-e`, `nrepl-server`, a `FILE`); the explicit subcommands below also work -anywhere: +`bin/joltc` reads a `deps.edn` in the current directory, fetches its +dependencies, and prepends the resolved source directories to the source roots +for the run. The CLI commands (`jolt.deps` + `jolt.main`): ```bash -jolt -M:test [args] # run the :test alias's :main-opts (the usual entry) -jolt -A:dev repl # run a command with the :dev alias's extra paths/deps -jolt run FILE [args] # resolve, then run FILE -jolt path # print the resolved roots (':'-joined) -jolt tasks # list :tasks from deps.edn -jolt task NAME [args] # run a task +bin/joltc run -m NS [args] # resolve deps.edn, load NS, call its -main +bin/joltc run FILE # resolve deps.edn, load a Clojure file +bin/joltc -M:alias [args] # run the alias's :main-opts +bin/joltc -A:alias [args] # add the alias's paths/deps, then run the rest +bin/joltc repl # start a line REPL +bin/joltc path # print the resolved source roots (':'-joined) +bin/joltc # run a deps.edn :tasks entry ``` -So, for example, to start an nREPL server that loads a project and its deps, -add `:aliases {:nrepl {:main-opts ["nrepl-server"]}}` to `deps.edn` and run -`jolt -M:nrepl` (or just `jolt nrepl-server`, which auto-resolves the `deps.edn`). - Example `deps.edn`: ```clojure {:paths ["src"] :deps {weavejester/medley {:git/url "https://github.com/weavejester/medley" - :git/tag "1.0.0"} + :git/sha ""} my/helpers {:local/root "../helpers"}}} ``` ```bash -jolt run -m myapp.main +bin/joltc run -m myapp.main ``` ### What's supported -- **git deps** — `{:git/url … :git/tag …}` or `{:git/url … :git/sha …}` (use a - full SHA; `git fetch` can't resolve a short one). Transitive deps from each - dependency's own `deps.edn` are resolved too. +- **git deps** — `{:git/url … :git/sha …}` (use a full SHA; `git fetch` can't + resolve a short one), with an optional `:deps/root` for a subdirectory. + Transitive deps from each dependency's own `deps.edn` are resolved too. - **local deps** — `{:local/root "../path"}`. - The project's own `:paths` (default `["src"]`) are included. - **aliases** — `:aliases {:dev {:extra-paths ["dev"] :extra-deps {…} :main-opts ["-e" "…"]}}`, selected with `-A:dev` (or several: `-A:dev:test`). `:extra-paths`/`:extra-deps` accumulate across selected aliases; `:main-opts` is last-wins and runs via `-M:alias`. -- **user config** — a `deps.edn` under `$JOLT_CONFIG` (else - `$XDG_CONFIG_HOME/jolt`, else `~/.jolt`) merges beneath the project's, the - way `~/.clojure/deps.edn` does: `:deps`/`:aliases`/`:tasks` merge per key - with the project winning. -- **tasks** — `:tasks {clean "rm -rf target" test {:doc "run the suite" - :main-opts ["-e" "(run-tests)"]}}`. A string task is a shell command; a map - task runs jolt with its `:main-opts`. `jolt tasks` lists, `jolt task NAME` - runs. +- **tasks** — `:tasks {clean "rm -rf target" test {:main-opts ["-m" "…"]}}`. + A string task is a shell command; a map task runs jolt with its `:main-opts`. + Run one with `bin/joltc `. -Conflicts resolve the tools.deps way: resolution is breadth-first, so a -top-level coordinate always beats a transitive one for the same lib, and -conflicting coordinates print a warning naming both. +Resolution is breadth-first, so a top-level coordinate always beats a transitive +one for the same lib. Git clones land in a global, sha-immutable cache shared across projects — -`$JOLT_GITLIBS`, else `/gitlibs` (the `~/.gitlibs` model). The -resolved roots are cached per project in `.cpcache/jolt-deps.jdn`, keyed on a -hash of the project `deps.edn` + the user `deps.edn` + the selected aliases. +`$JOLT_GITLIBS`, else `~/.jolt/gitlibs`. ### What's not -- **No Maven.** `:mvn/version` deps are ignored — git and local only. +- **No Maven.** `:mvn/version` deps are skipped with a warning — git and local + only. - **Pure `clj`/`cljc` only.** A library that needs the JVM (Java interop, host classes) or a `clojure.core` feature Jolt doesn't implement will fail to load or fail at a call. Coverage is per-function: a namespace can load with most functions working and a few not. -### Bundling into one file - -`jolt uberscript OUT.clj -m NS` bundles `NS` and every namespace it requires — -your code plus its dependencies — into a single `.clj` in dependency order, -ending with a call to `NS/-main`. Run it from a project dir and the `deps.edn` -is resolved first, so dependency namespaces are on the path to bundle. The -result runs on a plain `jolt` with no `JOLT_PATH`, no deps fetched, and no jpm: - -```bash -jolt uberscript app.clj -m myapp.main -jolt app.clj arg1 arg2 -``` - See [`tools-deps.md`](tools-deps.md) for the design rationale. diff --git a/docs/chez-phase0-results.md b/docs/chez-phase0-results.md deleted file mode 100644 index 4a1bd2a..0000000 --- a/docs/chez-phase0-results.md +++ /dev/null @@ -1,48 +0,0 @@ -# Chez port — Phase 0 results (jolt-cf1q.1) - -De-risk + contract harness. Done; all gates green. Decisions feed Phases 1–3. - -## 0a — value model (`host/chez/values.ss`, `test/chez/values-test.ss`) -Jolt value layer on Chez: nil sentinel (distinct from `#f`/`'()`), interned -keywords (NUL-separated intern key, no ns/name collision), ns+meta symbols, -exactness-aware `jolt=` ((= 1 1.0) is false), and a `jolt-hash` consistent with -it (non-finite-float safe). Chez's numeric tower IS Clojure's — ratios + bignums -come free. **37/37 tests.** - -## 0b — host-neutral contract gate (`test/chez/`) -The spec corpus is data, so one contract gates every host. Extracted 2655 -`[label expected actual]` cases from `test/spec/*.janet` into `corpus.edn` (valid -as BOTH EDN and Janet data). `run-corpus.janet` drives ANY jolt binary (pluggable -`JOLT_BIN`) at the CLI boundary, one fresh subprocess per case. Baseline vs Janet -`build/jolt` (compile mode, the port's target mode): **2641/2655**, 14 known CLI -divergences allowlisted (interpret-vs-compile leniency + invoke-collection-as-fn, -several non-canonical vs JVM anyway). **The gate fails only on NEW divergences** — -exactly what we want pointed at the Chez host in Phase 1+. - -## 0c — persistent-collection perf (`spike/chez/collections-experiment.ss`) -The shim-vs-self-hosted decision for collections. Map-churn workload from -`bench/collections.clj` (30000 assoc/get over 4096 keys), correct result (30000): - -| | mean | vs Janet | vs native ceiling | -|---|--:|--:|--:| -| Janet jolt HAMT | 258.6 ms | 1× | — | -| Chez persistent HAMT (hand-Scheme) | 6.3 ms (opt3) | **~41×** | ~15× | -| Chez native hashtable (mutable) | 0.43 ms | ~600× | 1× | - -**Decision: self-host the persistent collections in Clojure (jolt-core).** A -persistent HAMT on the Chez substrate is ~41× faster than Janet's, so the -substrate is not the bottleneck; a compiled-Clojure HAMT should land near the -hand-Scheme one (cf. the mandelbrot finding that Chez compiles emitted code to -the native ceiling). The ~15× gap to mutable-native is the inherent persistence -cost (node-copy per assoc), identical in kind to JVM Clojure, and closes with -transients/editable nodes when needed. Keep a Scheme-shim HAMT as fallback ONLY -if Phase 2 shows the compiled-Clojure version underperforms. - -Caveats (spike scope): the experiment uses integer-key-as-hash (shallow, -collision-free trie) and `merge-leaves` lacks real collision nodes — fine for the -substrate-speed question; the real RT needs `jolt-hash` + collision handling. - -## Net -Substrate speed (compute + collections), value model, and the parity gate are all -validated and green. Phase 1 can bootstrap the real pipeline against a known, -enforceable contract. diff --git a/docs/chez-port-plan.md b/docs/chez-port-plan.md deleted file mode 100644 index b0096c5..0000000 --- a/docs/chez-port-plan.md +++ /dev/null @@ -1,198 +0,0 @@ -# Re-hosting Jolt on Chez Scheme — phased plan - -Decision (2026-06-17): port jolt's runtime substrate from Janet to Chez Scheme -(cisco/ChezScheme). The spike (`spike/chez/RESULTS.md`) validated the thesis: -the compute-substrate ceiling is ~12-47x faster than Janet (mandelbrot -166->13.4ms matching jolt's own C-codegen result; fib 246->5.2ms), Chez's -native compiler reaches that at runtime with the REPL intact, size stays -single-digit MB, and the one regression (memory baseline ~2.5x) is opt-in -tunable via WPO + stripped boot + AOT-under-petite. - -This plan is built around two north stars beyond raw speed: - -1. **Zero Janet — Chez is the sole substrate** (revised 2026-06-18). The goal is - not a minimal Janet shim that coexists with Chez; it is to rip Janet out - entirely and rely on Chez going forward. Two things must move off Janet: the - **runtime** (a hand-written Chez Scheme RT replaces the Janet value layer / - vars / evaluator) and the **compiler**. The compiler is the subtle part: the - analyzer/IR are already portable Clojure, but today they *execute on the Janet - host*, and the IR->Scheme emitter + driver are *Janet code* (`host/chez/ - emit.janet`), so the current `clojure.core` prelude is a Janet cross-compile. - The end state requires Chez-jolt to run the analyzer itself and the emitter to - become portable Clojure — so Chez-jolt compiles its own `clojure.core` AND the - analyzer from source, with no Janet in the loop (the bootstrap fixpoint). Then - both `src/jolt/*.janet` and `host/chez/*.janet` are deleted. Every line that - stays in Scheme is hand-written Chez RT, not Janet; the forcing function for - jolt-tzo/uqi/lcn still applies (push logic into `jolt-core/`). -2. **Tests are the contract.** The spec/conformance corpus is host-neutral data - (`[name expected-clj actual-clj]` triples compared via jolt's own `=`). It is - the acceptance gate for "the port is correct" — Chez-jolt must pass the same - corpus the Janet host passes, with no regression to the clojure-test-suite - baseline. - -## The Chez host RT vs portable Clojure (target end-state) - -What MUST be hand-written Chez Scheme (the irreducible primitive layer the -self-hosted core rests on) vs what MOVES into portable Clojure. Nothing here is -Janet — this is the split *after* Janet is gone: - -### Stays in Scheme (the Chez host RT) -- **Value primitives that can't bottom out in Clojure without circularity:** - the `nil` sentinel (distinct from `#f` and `'()` — the classic Lisp-on-Lisp - trap), keyword/symbol records (Clojure symbols carry ns + meta), char/string - bridging. NOTE: Chez's **numeric tower is a windfall** — int/float/ratio/ - bignum are native, so jolt gets exact ratios + bignums for free (Janet lacked - them). -- **Mutable cell / box + array primitives** the self-hosted RT builds on (var - roots, transients, HAMT node arrays). -- **A type-tag / record primitive** for deftype/protocol dispatch (Chez records). -- **`host/compile`** — eval a backend-emitted Scheme form to a procedure. On Chez - this is literally `eval`/`compile`. Trivial. This is the whole backend - host-dependency. -- **FFI / interop bridge** (foreign-procedure) for host interop calls. -- **Persistent-collection hot nodes** — ONLY IF the Clojure-on-Chez version - (Phase 0c) doesn't hold perf. Open question, decided by measurement. - -### Moves into portable Clojure (`jolt-core/`) -- **The reader** (text -> forms). CLJS self-hosts its reader; ours can too. ~33KB - of Janet leaves the host. Not hot. -- **Analyzer + IR + passes** — already portable Clojure source; the change is - that they must EXECUTE on Chez-jolt, not on the Janet host (Phase 3). -- **The backend emitter** — today it is Janet (`host/chez/emit.janet`); its LOGIC - becomes portable Clojure (`jolt.backend-scheme`) that emits Scheme forms as - data, so it runs on Chez. Only `host/compile` (Chez `eval`) crosses the seam. -- **macros + clojure.core** — finish the jolt-uqi/tzo migration (most already - Clojure). -- **Protocol/multimethod dispatch logic** — over the host tag primitive. -- **Persistent collections** — candidate (Phase 0c), perf-permitting. - -### Dropped entirely -- **The tree-walking interpreter** (`eval_base/eval_runtime/eval_special/ - eval_resolve`, ~140KB Janet). On Chez, native `compile` is always present and - cheap, so the compile-only path can cover every form — no `jolt/uncompilable` - fallback needed. The interpreter's role as the correctness *oracle* transfers - to the spec corpus + JVM Clojure (the real reference), which is strictly - better. This is the single largest shim reduction and the biggest open risk; - Phase 1 validates that compile-only is total before we commit to the drop. - -## Test contract strategy - -- **Corpus is the contract.** The 44 `test/spec/*.janet` files, the conformance - cases, `clojure-test-suite`, and `clojure-stdlib-suite` are host-neutral: pure - Clojure source + expected values. Extract the triples into a runner that can - target an arbitrary `jolt` binary (subprocess at the Clojure boundary). -- **Parity gate.** Chez-jolt must pass the same corpus as Janet-jolt; the - clojure-test-suite baseline is the bar (raise it when it rises, never lower). -- **Oracle shift.** Today's 3-mode conformance (interpret/compile/self-host) - loses the "interpret" leg when the interpreter is dropped; the golden expected - values + JVM Clojure become the oracle. Keep the frozen expected values. -- **Dual-run during migration.** Run BOTH hosts against the corpus until Chez - reaches parity, then retire the Janet host. - -## Phases (-> beads epic) - -**Phase 0 — Foundations & contract harness** (de-risk; no jolt pipeline yet) -- 0a. Chez RT value model: nil sentinel, keyword/symbol records, numeric-tower - mapping, `=`/hashing. Resolve the nil/`'()`/`#f` representation up front. -- 0b. Host-neutral test-contract runner: extract the spec/conformance corpus to - drive an arbitrary jolt binary; stand up the parity-gate machinery. -- 0c. Persistent-collection perf experiment: HAMT/PV in Clojure-on-Chez vs - Scheme-native — the data that decides what stays shim vs self-hosted. - -**Phase 1 — Minimal Chez kernel + real-pipeline bootstrap** -- Scheme shim (value layer, var/ns cells, `host/compile`, cenv impl). -- `jolt.backend` Scheme-emit target for the IR the analyzer already produces. -- Bootstrap jolt-core (ir/analyzer) on Chez; compile + run `(+ 1 2)` -> fib -> - mandelbrot through the REAL pipeline. Gate: compute benches run end-to-end and - hit ~the spike ceiling; confirm compile-only is total (no fallback needed). - -**Phase 2 — clojure.core to spec parity** -- Bring up persistent collections (per 0c) + seq/coll/print/refs/io tiers over - the Chez RT. Gate: spec + conformance + clojure-test-suite parity with the - Janet baseline. - -**Phase 3 — Self-host the compiler on Chez** (the no-Janet spine) -- Rewrite the IR->Scheme emitter from Janet (`host/chez/emit.janet` + `driver. - janet`) into portable Clojure in jolt-core (a `jolt.backend-scheme` target); - folds jolt-lcn. Move the reader into jolt-core. Stand up Chez compile-from- - source: Chez-jolt reads the `.clj` tiers, runs the analyzer *executing on Chez* - (not Janet), emits Scheme, evals — replacing the Janet cross-compile of the - prelude. Bootstrap fixpoint: Chez-jolt compiles `clojure.core` AND the analyzer - from source with no Janet in the loop; verify stage2==stage3 emitted forms. - Drop the tree-walking interpreter. Continue core-* leaf migration (jolt-uqi/ - ded/tzo). Gate: Chez-jolt builds itself from source, full corpus parity holds, - zero Janet invoked. - -**Phase 4 — Deployment & optimization modes** (the "optimize specific cases" lever) -- Wire `JOLT_WHOLE_PROGRAM`/direct-link to emit specialized Scheme (fl*/fx*), - feeding Chez `compile-whole-program`. `jolt build`: WPO + strip-fasl + - AOT-under-petite + heap tuning -> small fast binary (jolt-0w9u reframed). - Rebuild fibers/async on call/cc + threads. Gate: full bench suite incl. - collections/binary-trees (the GC axes); size + memory measured vs spike - baseline. - -**Phase 5 — Delete the Janet host** (single substrate) -- Chez self-hosts + parity + perf confirmed -> delete both `src/jolt/*.janet` - (the seed) and `host/chez/*.janet` (the Janet emitter/driver, now superseded by - the Clojure `jolt.backend-scheme`). Chez is the only substrate; no alternate - host retained. jolt-core unchanged. Oracle stays the spec corpus + JVM Clojure. - Net: one host, no Janet, no cross-compile. - -## Host interop & the examples acceptance corpus - -The `../examples` repo holds real jolt apps with real git deps + C interop; they -are the **end-to-end acceptance gate** complementing the unit-level spec corpus. -The port must account for jolt's interop surface, which is layered: - -- **`janet.*` bridge** — the general Clojure->Janet escape hatch (`janet/get`, - `janet/struct`, `janet.ev/sleep`, `janet.net/close`, `janet.spork.http/*`), - used mostly in demo glue. DECIDED (2026-06-17): rename the bridge to a - **neutral `host.*`** namespace (not `janet.*`/`chez.*`), so app interop code is - host-portable; each host implements `host.*` over its own FFI (Janet FFI today, - Chez `foreign-procedure` tomorrow). The legacy `janet.*` aliases stay as - deprecated shims during migration. -- **FFI-backed shim libraries** — the genuine C interop. `jolt-lang/http-client` - implements `java.net` + TLS + gzip as host shims **over Janet FFI**, backing - clj-http-lite's `:clj` branch. On Chez these are reimplemented over Chez FFI - (libcurl/openssl/zlib or native sockets); the `java.*`-facing API is unchanged. - Also in scope: `jolt-lang/db`, `jolt-lang/logging`, and `jolt-lang/router` - (mirrors `reitit.Trie`; verify whether it carries native code). -- **`:jpm/module` Janet-native deps** — `spork/http` (server), `ring-janet- - adapter`. No Chez equivalent: provide a Scheme/Chez-FFI HTTP server or treat - these as test-only fixtures. -- **Native-library dependencies in deps.edn** (DECIDED 2026-06-17). C-interop - shims need shared libraries (libcurl, openssl/libssl, zlib) that CANNOT be - pulled from git like Clojure libs. Add a `:native` dep form so a project can - *declare* what it needs, e.g. - `{:native/lib "curl" :native/min-version "7.0" :native/header "curl/curl.h"}`. - The resolver doesn't fetch them, but: (a) it surfaces the requirement to the - user (and can suggest the install command per platform), and (b) the - loader/`foreign-procedure` layer probes for the `.so`/`.dylib` at load and, if - absent, raises a precise error — "missing native library: libcurl (declared by - jolt-lang/http-client); install with `brew install curl`" — instead of a raw - dlopen failure. Symmetric with `:jpm/module`'s "verify importable, hint to - install" pattern. -- **Host-neutral pieces that port for free** — Java-class mirror shims (pure - Clojure) and `JOLT_FEATURES` reader conditionals. - -Sequencing: the interop bridge mechanism is part of the Phase-1 shim, but the -FFI-backed libraries land in **Phase 4** (downstream of core parity), validated -by running the examples end-to-end. Tracked as a dedicated epic child. - -## Beads reconciliation - -- **Closed (obsolete — Janet bytecode-VM / cgen / Janet-dispatch mechanisms Chez - replaces wholesale):** jolt-ffn (epic, already concluded flat), jolt-5vsp.1, - jolt-qx70, jolt-l1l4 (cgen), jolt-cm7t, jolt-fw2 (Janet dispatch substrate), - jolt-pria (Janet ctx cold-build startup). -- **Recommend close (confirm) — Janet constant-factor passes Chez's JIT/GC/WPO - subsume:** jolt-826, jolt-27w, jolt-t6r, jolt-8flj, jolt-3ko, jolt-t34, - jolt-u1f. (jolt-ffn's own STATUS says the gap is "generic-runtime overhead… - the JVM erases via JIT + inline caching + unboxing" — exactly Chez's job.) -- **Reframed under the Chez epic:** jolt-0w9u + .1 -> Phase 4 deploy mode + - closed-world audit; jolt-1r86 -> Phase 0b/4 bench validation; jolt-lcn / - jolt-uqi / jolt-tzo / jolt-ded / jolt-brh / jolt-7dl -> Phase 2/3 self-hosting, - now targeting Chez; jolt-5vsp (foundational-runtime epic) -> parent, the Chez - port is its realization. -- **Keep host-agnostic:** deps beads (jolt-x4o, jolt-xkd, jolt-pnje, jolt-vley); - correctness bug jolt-jk23. diff --git a/docs/foundational-runtime-handoff.md b/docs/foundational-runtime-handoff.md deleted file mode 100644 index 31f123f..0000000 --- a/docs/foundational-runtime-handoff.md +++ /dev/null @@ -1,198 +0,0 @@ -# Foundational Runtime Epic — Handoff - -**Epic:** jolt-5vsp · **Predecessor:** jolt-ffn (targeted specialization — concluded) -**Date:** 2026-06-16 - -This is a cold-start handoff. Read it top to bottom before touching code. Its -whole point is to keep the fresh session from re-running the experiments that -already came back flat, and to start from the one measurement that actually -tells us where to invest. - -## Why this epic exists - -The targeted-specialization epic (jolt-ffn) tried to close jolt's constant-factor -gap vs JVM Clojure with per-form compiler passes. Three independent attempts all -came back flat: - -| Attempt | Bead | Result | -|---|---|---| -| Record field-read guard removal (bare field reads) | jolt-3ko | ~3% on dispatch (shipped #141 — kept for correctness, not speed) | -| Protocol inline cache (runtime, per-method) | jolt-ez5h | ~0% — the per-dispatch gen-check exactly cancels the find-protocol-method saving; `find` was never the bottleneck | -| Record-ctor descriptor-baking (fewer allocs/record) | jolt-p7fo | flat on binary-trees + broke the gate; reverted | - -The conclusion: **the gap is structural to jolt-on-Janet, not a missing -optimization.** Targeted passes remove only the cheap parts; the structural floor -remains. - -## The scorecard (jolt / JVM Clojure) - -Regenerate any time with `JVM=1 bench/run.sh` (the absolute-reference mode). - -| Axis | Bench | jolt/JVM | -|---|---|---| -| Pure float compute | `mandelbrot` | **~15× ← THE FLOOR** | -| Persistent collections (HAMT) | `collections` | ~28× | -| Recursion (call + arith) | `fib` | ~37× | -| Megamorphic dispatch | `dispatch` | ~76× | -| Monomorphic dispatch | `mono-dispatch` | ~109× | -| Allocation / GC | `binary-trees` | ~314× (≈150× at depth 10) | - -`mandelbrot` is the floor: pure tight arithmetic loops — no dispatch, no -allocation, no collections — and native arith already fires (jolt-3pl). So ~15× -is what jolt's *execution substrate* costs on the simplest possible workload. -Every other axis adds structural overhead **on top** of that floor. - -**Machine caveat:** the dev machine swaps heavily (~13 GB). Alloc-heavy benches -(`binary-trees`, `collections`) inflate badly; light benches (`mandelbrot`, -`fib`, `dispatch`) are trustworthy. Get absolute alloc numbers on a clean machine. - -## The four structural walls - -1. **Bytecode-VM execution.** jolt's backend emits **Janet** (a register-bytecode - VM) and runs it on the Janet interpreter loop — no JIT, no native code. Every - op is bytecode dispatch. This is the `mandelbrot` 15× floor. -2. **Mark-sweep GC.** Janet's GC scans all live objects each cycle (no - generations). Live-data + alloc-heavy workloads (`binary-trees` retains the - tree) pay O(live) per GC. The JVM's generational GC makes young-object churn - nearly free. -3. **Indirect calls.** Protocol dispatch and fn calls go through indirection - (closures, the protocol registry). The JVM inlines/devirtualizes. jolt's - devirt (jolt-41m) only fires on *statically*-proven monomorphic sites; - `reduce`/`mapv` over a collection doesn't give that proof, so the common - runtime-monomorphic case pays full dispatch (that's why `mono-dispatch` is - *worse* than megamorphic — the JVM inline-caches it to near-free, jolt doesn't). -4. **Boxed / generic representations.** Records are tuples `[descriptor field…]`; - field access goes through a tag guard unless the type is proven. Generic ops - carry runtime type checks. (Open question: are Janet *numbers* boxed? Verify - in the spike — it decides whether unboxing is a lever or already done.) - -## Foundational levers (ranked) - -1. **Native codegen — emit C, not Janet bytecode.** The Stalin approach. Compile - jolt IR → C → machine code via the system compiler. The *only* lever that - moves the 15× compute floor; could approach C/JVM speed on compute-bound code. - Massive (a new backend). Plausible incremental shape: a jolt-IR→C compiler for - *hot* fns with a fallback to the existing bytecode path for unsupported forms — - mirroring today's interpret/compile hybrid. Needs to confirm Janet's C-API / - native-module story can be targeted incrementally. -2. **Structural GC-pressure reduction.** Value-type small records (avoid heap), - transient/editable-node hot paths (RFC 0003 future work — pvec/phm/sorted are - now tries/HAMT/RB, so O(1) `transient`/`persistent!` via editable nodes is - open). Helps the alloc-bound axes (`binary-trees`, `collections`). Does **not** - touch the compute floor. -3. **Deeper devirt + body inline.** Propagate element/return types so devirt - fires on runtime-monomorphic collections, then inline the method body - (jolt-4x9 element types + jolt-t6r). Helps dispatch. Bounded ceiling (still - bytecode underneath). - -## STATUS (2026-06-16) — lever 1 (native codegen) built and working - -The spike ran and lever 1 is now implemented. Full writeups: -`docs/foundational-runtime-spike-results.md` (floor localization) and -`docs/foundational-runtime-lever1-native-codegen.md` (native codegen). - -Done (all merged to main, PRs #143–#148): -- **Floor localized:** the 15.4× decomposes into a **Janet-VM floor ≈10.8× JVM** - (only native codegen moves it) + a **jolt loop-lowering ≈1.43×** (cheap backend - win, bead **jolt-v28u**). Janet numbers are already unboxed (not a lever). -- **Native codegen (jolt-ihdp, CLOSED):** `src/jolt/cgen.janet` translates - numeric-leaf fns (numeric in/out, native-op arithmetic + loop/recur/if/let/do) - to C. Wired into the backend `:def` emit under **`JOLT_CGEN=1`** (opt-in). The - `.so` is content-addressed + cached. **mandelbrot 224ms → 12.4ms (~18×)**, - beats JVM. Leaf-first falls out free (callers stay bytecode, call native fn). -- **Build-time AOT (jolt-a7ds, partial):** `:cgen-collect?` records leaf fns at - build, `aot-build` compiles them into one `.so` + manifest; `:cgen-prebuilt` + - `load-aot` install them at deploy with **no cc** (proven with cc off PATH). - -Open work under epic jolt-5vsp: -- **jolt-a7ds** — fuse the prebuilt `.so` + manifest into the `jpm` exe for a - literal single binary (+ a `jolt cgen-build -m app` CLI). The heaviest piece; - into jpm executable-build, not the compiler. -- **jolt-v28u** — `while`-loop lowering for tail `recur` (cheap ~30%, independent - of cgen; helps ALL loops, not just cgen candidates). -- **jolt-l1l4** — widen cgen numeric grammar (mod/rem/bit-ops/min/max, mixed fns). -- **jolt-qx70** — hot-fn auto-detection (drop the global `JOLT_CGEN` knob). -- Lever 2 (GC-pressure) and lever 3 (deeper devirt) — untouched; see below. - -The original spike instructions are preserved below for context. - -**Localize the 15× floor.** Build three `mandelbrot` implementations and compare: - -- **jolt-compiled** `mandelbrot` (already in `bench/mandelbrot.clj`), -- **hand-written Janet** `mandelbrot` (the same nested loop, idiomatic Janet — - write it directly, no jolt), -- **JVM Clojure** `mandelbrot`. - -Two ratios fall out: - -- **jolt-emitted-Janet vs hand-Janet** → how much overhead jolt's *backend* adds - over optimal Janet. To see jolt's emitted Janet, use the backend emit path - (`backend/emit-ir` on the analyzed `run`/`count-point` fns) — note `:arities` - etc. are jolt pvecs, so introspection is awkward; easier to read the emitted - Janet via the compile path or just A/B the timings. -- **hand-Janet vs JVM** → the Janet VM's own floor. - -Decision: - -- If **hand-Janet ≈ jolt** and hand-Janet is ~15× JVM → the floor is **Janet's - bytecode VM**. Native codegen (lever 1) is the only fix. Commit to the spike of - a jolt-IR→C path for one hot fn and measure. -- If **jolt ≫ hand-Janet** → jolt's backend emits suboptimal Janet; there's - headroom in the **backend** (cheaper, no new runtime). Find what it emits that - hand-Janet doesn't. - -Also measure the **GC share** on `binary-trees` (Janet GC stats around the run — -`(gccollect)` / `gcinterval`, or count allocations) to size lever 2 honestly. - -## Key files / mechanisms - -- **Backend (IR → Janet emit):** `src/jolt/backend.janet`. `native-ops` (~L322) - emits native Janet arith; `emit-ir` (~L674) runs passes then emits. A native-C - backend would branch here. -- **Passes / inference:** `jolt-core/jolt/passes.clj` (`run-passes`), - `jolt-core/jolt/passes/types.clj` (inference; the `:fn` branch ~L527 now seeds - ^Record param hints — #141), `jolt-core/jolt/passes/inline.clj` - (scalar-replace, `ctor-shape`). -- **Record representation:** `src/jolt/types_protocols.janet` — `make-record` - (~L145, the ~5-alloc/record path), `record-shape-for` (~L139, rebuilds its - cache key every call), `record-tag`. Records are tuples `[descriptor field…]`. -- **Dispatch + ctors:** `src/jolt/eval_runtime.janet` — - `protocol-dispatch-impl` (~L62), `make-deftype-ctor-impl` (~L382). -- **Config knobs:** `src/jolt/config.janet` — `JOLT_DIRECT_LINK`, - `JOLT_WHOLE_PROGRAM`, `JOLT_OPTIMIZE`, the `ctx-shaping-env-vars` list (any new - ctx-shaping env var MUST be added there and to `image-cache-path`). -- **Self-hosting design:** `docs/self-hosting-compiler.md` (the kernel/value-layer - boundary), `docs/rfc/0003-transients.md` (editable-node future work). - -## How to build, run, measure - -```sh -jpm build # build/jolt (ctx baked, ~20ms startup); from-source is ~8s cold -export PATH="$PWD/build:$PATH" -bench/run.sh # jolt only, WP on -JVM=1 bench/run.sh # jolt vs JVM scorecard (needs `clojure` on PATH) -bench/run.sh mandelbrot 400 # one bench, custom size -JOLT_WHOLE_PROGRAM=0 bench/run.sh # measure what WP buys -``` - -Gate: `jpm build; janet run-tests.janet` (parallel, ~100s; `JOLT_TEST_JOBS` -overrides). Bench memory hygiene (`bd memories bench-isolation-gotcha`): never run -a perf matrix while other CPU work runs — it starves later configs and produces -bogus numbers. Sandwich A/B/A. - -## What NOT to repeat (already flat — see beads for detail) - -- Runtime protocol inline cache (jolt-ez5h): gen-check cancels the saving. -- Field-read guard removal as a *speed* play (jolt-3ko): ~3%; machinery dominates. - (The #141 change is kept for correctness + the `with-meta`-on-symbols fix.) -- `make-record` descriptor-baking (jolt-p7fo): flat — `binary-trees` is dominated - by the live retained tree + GC, not the short-lived intermediate allocs. - -## Open questions for the spike - -- Are Janet numbers boxed? (Lever or already done.) -- Does Janet expose a native-module / C-codegen path jolt can target incrementally - (hot fns → C, rest → bytecode)? -- What fraction of `binary-trees` is GC vs execution? -- Is there a cheaper record representation (Janet struct vs tuple-with-descriptor) - that lowers field-read + alloc cost without a new backend? diff --git a/docs/foundational-runtime-lever1-native-codegen.md b/docs/foundational-runtime-lever1-native-codegen.md deleted file mode 100644 index a037d7e..0000000 --- a/docs/foundational-runtime-lever1-native-codegen.md +++ /dev/null @@ -1,173 +0,0 @@ -# Lever 1 — Native codegen (jolt-IR → C): feasibility spike - -**Epic:** jolt-5vsp · **Date:** 2026-06-16 -**Predecessor:** the localization spike (`docs/foundational-runtime-spike-results.md`) -showed the 15.4× mandelbrot floor is ~70% Janet-VM floor (only native codegen -moves it) + ~30% loop-lowering (cheap backend fix, jolt-v28u). This spike probes -**lever 1's ceiling and the incremental hot-fn-in-C strategy** before committing -to a backend. - -All legs return the identical result (3288753 at n=200). Numbers are means of 3 -after warmup; the dev machine swaps, so treat these as orders-of-magnitude (the -≈ vs JVM call is robust; ±2ms is noise). - -## The native-C ceiling — it beats JVM - -Native mandelbrot built as a Janet native module (`spike/native/mandel.c`): - -| Leg | mean | vs jolt (219ms) | vs JVM (14.2ms) | -|---|---|---|---| -| **native-C whole run** (pure C, no Janet in loop) | **~10–12 ms** | **~18–22× faster** | **faster than JVM** | -| Janet loop → C hot-fn (forward crossing) | ~11–13 ms | ~18× faster | ≈ JVM | -| C loop → `janet_call` bytecode (reverse crossing) | ~152 ms | ~no better | ~11× slower | -| *(reference)* jolt-compiled | 219 ms | — | 15.4× | -| *(reference)* JVM Clojure | 14.2 ms | — | 1.0× | - -**Verdict: lever 1 is validated and its ceiling is excellent.** Compiling the hot -compute path to C makes it ~18–22× faster than today's jolt and *edges out JVM -Clojure* — native code has no VM-dispatch floor at all. This is the only lever -that touches the ~10.8× Janet-VM floor, and the payoff is the full gap. - -## The crossing-direction rule (the key strategic finding) - -The boundary cost is wildly asymmetric: - -- **Forward (bytecode → C): nearly free.** A Janet bytecode loop calling a C - hot-fn n² (=40 000) times runs at ~11–13 ms — within ~15% of pure C. So you can - compile just the *inner* hot fn to C and capture ~95% of the win while the outer - loop stays bytecode. **Incremental adoption works.** -- **Reverse (C → `janet_call` → bytecode): ~3.5 µs/call.** A C fn calling a - bytecode helper per iteration runs at ~152 ms — *no better than jolt today*. The - `janet_call` cost (entering the VM/fiber per call) dominates. - -**Design constraint → compile leaf-first / whole-hot-cluster.** A fn is a -profitable C-compilation candidate only if its hot path calls **nothing that stays -in bytecode** — only primitives or other C-compiled fns. Cross the boundary only at -*cold* edges. For mandelbrot, `count-point` is a leaf (calls only arithmetic -primitives) → the ideal first target; compiling it alone captures the win -(forward crossing), but a half-compiled hybrid that `janet_call`s back per -iteration buys nothing. - -## The dynamic-compile path works (no jpm needed) - -jolt's compile model is dynamic (analyze → IR → Janet → eval at runtime). Native -codegen fits the same shape: a `.so` compiled with a **plain `cc` invocation** -(no jpm/project.janet) loads at runtime via `require` and runs at full native -speed (verified: `run-c(200)` correct, 13.5 ms cold). - -``` -cc -shared -fPIC -O2 -I/opt/homebrew/include -undefined dynamic_lookup \ - mandel.c -o mandel.so # macOS; Linux drops -undefined dynamic_lookup -(require "path/to/mandel") # loads at runtime, cfunctions callable -``` - -So the native tier mirrors today's interpret/compile hybrid: emit C for a hot -fn → shell to `cc` → `require` the `.so` → bytecode callers call into it via the -(cheap, forward) native-module call path. Caching keyed by fn-source-hash mirrors -the existing ctx image cache. - -## Toolchain confirmed (this machine) - -- `janet.h` present (`/opt/homebrew/include/janet.h`, Janet 1.41.2). -- `jpm declare-native` builds a `.so` cleanly. -- Direct `cc` (no jpm) builds a loadable `.so`. -- C API used: `janet_getnumber/getinteger`, `janet_wrap_number`, `janet_fixarity`, - `janet_getfunction`, `janet_call`, `janet_cfuns`, `JANET_MODULE_ENTRY`. - -## Status: wired into the compile path (JOLT_CGEN, opt-in) - -`src/jolt/cgen.janet` (IR→C translator) is wired into the backend's `:def` emit -via `cgen-root`, gated behind **`JOLT_CGEN=1`** (off by default; needs -direct-linking). When on, a plain defn of a numeric-leaf fn is compiled to C at -def time and the cfunction installed as the var root — so direct-linked callers -embed native code. The fn is NOT inline-stashed when cgen fires (callers must -call the C fn, not inline the bytecode body). `^:redef`/`^:dynamic` defns stay -bytecode. - -The leaf-first rule emerges for free: `run` calls `count-point` (a user var, not -a native-op), so `run` isn't a numeric leaf and stays bytecode — calling the -native `count-point` over the cheap forward crossing. - -**Measured end-to-end (`jolt -m mandelbrot 200`): 224 ms → 12.4 ms, ~18×**, with -the correct result — matching the spike's native-C ceiling. The default gate -(cgen off) is unchanged. Tests: `test/integration/cgen-pipeline-test.janet`. - -Known limitation: building *core* with `JOLT_CGEN=1` would try to cgen core -numeric-leaf fns into the cached ctx image, where embedded cfunctions may not -serialize — keep cgen for app/user code until image-cache interaction is handled. - -## Build-time AOT: native speed without a toolchain on the target (jolt-a7ds) - -The JIT path above runs `cc` at runtime. The AOT path moves compilation to build -time so the deploy target needs no `cc`/`janet.h`: - -- **Build phase** (`:cgen-collect?`, needs cc): loading the app records every - numeric-leaf defn's IR; `cgen/aot-build` compiles them all into ONE native - module (`gen-c-module`) and `write-manifest` persists `{sopath, [{ns name sym}]}`. -- **Deploy phase** (`:cgen-prebuilt`, NO cc): `cgen/load-aot` loads the prebuilt - `.so` (via the `native` builtin — no compiler) into a qname→cfunction map; the - backend's `:def` hook installs each as the var root with the same timing as the - JIT path, so callers direct-link to native code. - -**Proven** (`spike/native/aot-demo.janet`, two processes): build with cc, then -deploy with `cc` removed from PATH → `count-point` is still native, mandelbrot = -3288753 at **12.4 ms** (full 18×). Test: `test/integration/cgen-aot-test.janet`. - -This removes the runtime-toolchain dependency — the core of the deployment story. - -### The literal single binary (`jolt cgen-build`, done) - -`src/jolt/cgen_build.janet` + the `jolt cgen-build -m NS -o OUT` CLI fuse the -native code into the executable, so an app ships as ONE static file — no sidecar -`.so`, no toolchain to run. The driver: - -1. loads the app with `:cgen-collect?` to get its numeric-leaf fns + the source - files loaded (the uberscript-style bundle); -2. emits `cg.c` (one native module of those fns via `cgen/gen-c-module`) + a - positional manifest; -3. stages a build dir: `src`/`jolt-core` symlinks into the jolt tree, `cg.c`, the - app bundle, and an entry that bakes the runtime, installs the native fns as var - roots (`:cgen-prebuilt`), and runs `-main`; -4. runs `jpm build` there — `declare-native` builds `cg.a`, `declare-executable` - static-links it into the final exe (jpm's `create-executable` marshals the - module's cfunctions and calls its static entry at startup). - -Build needs `cc` + `jpm`; the result needs neither. Proven end-to-end: -`test/integration/cgen-build-test.janet` builds the mandelbrot fixture, runs it -from a clean dir with no `src/` and no `cg.so`, and gets the right total at native -speed (the count-point leaf is the linked cfunction). - -Build mechanics that bit (codified in `cgen_build.janet`): `stdlib_embed` slurps -`.clj` relative to cwd, so the build runs in a dir mirroring the repo layout (the -symlinks); jpm hardcodes `./project.janet` and sets `syspath = modpath`; the -executable's dofile imports `cg` and static-links `cg.a`, neither ordered nor -release-built by default, so the deps are wired explicitly; cleanup must `lstat` -(never follow the tree symlinks). The inner build runs `--workers=1` so it doesn't -saturate cores inside the parallel test gate. - -Open follow-ups (filed): widen the cgen grammar (jolt-l1l4) so more of an app's -hot fns qualify; hot-fn auto-detection (jolt-qx70) to drop the manual collect; -DCE on the bundled source (the uberscript path already does it). - -## Open questions for the implementation (next beads) - -1. **IR→C for the numeric subset.** Translate jolt IR → C for proven-double - arithmetic + tail `loop`/`recur` (count-point's shape). The native-arith type - proof (jolt-3pl) that already gates native *Janet* arith is the same proof that - gates C unboxing — reuse it. Start narrow: unbox doubles at entry, primitive - ops inline, rebox at exit; bail to bytecode for any unsupported form. -2. **Boundary policy.** Non-primitive args stay Janet values (no unbox); - per-iteration calls allowed only to other C-compiled fns. Encode the - leaf-first/cluster rule as the compile-candidate predicate. -3. **Trigger + cache.** AOT at build/first-run vs lazy JIT on hot fns; `.so` - cache keyed by source hash + flags (add to `ctx-shaping-env-vars` / - image-cache machinery if it becomes a ctx knob). -4. **Coverage.** Closures/upvalues, multi-arity, `recur` across the C boundary, - portability of `cc` flags per platform. - -## Artifacts (`spike/native/`) - -- `mandel.c` — native mandelbrot: `run-c` (pure C), `count-point-c` (leaf cfn), - `run-callback` (C loop → `janet_call` back, the reverse-crossing probe) -- `project.janet` — `declare-native` build -- `bench-native.janet` — the three-leg benchmark + harness diff --git a/docs/foundational-runtime-spike-results.md b/docs/foundational-runtime-spike-results.md deleted file mode 100644 index 6bdb242..0000000 --- a/docs/foundational-runtime-spike-results.md +++ /dev/null @@ -1,92 +0,0 @@ -# Foundational Runtime Spike — Results (the 15× floor, localized) - -**Epic:** jolt-5vsp · **Date:** 2026-06-16 -**Spike:** the START HERE section of `docs/foundational-runtime-handoff.md` — -jolt vs hand-written-Janet vs JVM `mandelbrot`, to localize the ~15× compute -floor before committing to native codegen (lever 1) vs a backend fix. - -## Setup - -Three implementations of the same nested mandelbrot loop, all returning the -identical result (3288753 at n=200, confirming correctness across all legs): - -- **jolt-compiled** — `bench/mandelbrot.clj` (`jolt -m mandelbrot 200`, WP + direct-link on) -- **hand-Janet (`while`)** — `bench/mandelbrot-hand.janet` (idiomatic Janet: `while` + `var`/`set`) -- **JVM Clojure** — `bench/mandelbrot.clj` on the JVM - -Plus a diagnostic fourth leg: - -- **hand-Janet (recursive)** — `bench/mandelbrot-hand-rec.janet`: hand Janet that - *mirrors jolt's loop lowering* (self-recursive local closure called per - iteration), to test whether the loop lowering alone explains jolt's overhead. - -Numbers are stable and sandwiched (A/B/A/B); machine noise < 1%. - -## The numbers (n=200, mean of 3, after warmup) - -| Leg | mean | × JVM | -|---|---|---| -| JVM Clojure | 14.2 ms | 1.0× | -| **hand-Janet (`while`)** | **153.4 ms** | **10.8×** | -| hand-Janet (recursive, mirrors jolt) | 215.3 ms | 15.2× | -| **jolt-compiled** | **219.0 ms** | **15.4×** | - -## What this localizes - -The 15.4× floor **decomposes into two distinct layers**: - -1. **Janet VM floor ≈ 10.8× JVM** (70% of the gap). Optimal hand-written Janet — - pure `while` loop over unboxed doubles, zero allocation — is still ~11× slower - than JVM Clojure. This is the cost of the Janet bytecode VM itself (no JIT, no - native code). **Only native codegen (lever 1) can touch this.** It is the - dominant share and validates lever 1 as the big structural lever. - -2. **jolt backend loop-lowering ≈ 1.43× on top** (the remaining 30%). jolt is - `219 / 153 = 1.43×` slower than optimal Janet. The diagnostic leg pins this - *entirely* to one cause: jolt lowers every `loop`/`recur` to a **self-recursive - local closure called once per iteration**, not a `while` loop. Hand-Janet - written that same way (recursive leg) lands at **215 ms ≈ jolt's 219 ms** — - so the recursive-closure lowering accounts for essentially all of jolt's - backend overhead on pure-compute code. - - See the emitted Janet (`bench/dump-mandelbrot-emit.janet`): `emit-loop` - (`src/jolt/backend.janet:210`) produces - `(do (var L nil) (set L (fn (i zr zi) … (L (+ i 1) …))) (let (…) (L …)))` - and `emit-recur` (`:228`) produces the per-iteration call `(L …)`. It relies - on Janet TCO for stack safety, but each iteration still pays a function - invocation (frame setup + arg bind) that a `while` loop skips. - -## Decision - -The handoff posed it as binary (Janet-VM floor *or* backend headroom). It is -**both**, now sized: - -- **Native codegen (lever 1) is the only thing that moves the dominant ~70%.** - Confirmed as the big lever. Pursue the incremental jolt-IR→C spike for one hot - fn next, per the handoff. -- **A cheap, localized ~30% win sits in the backend**, independent of any new - runtime: lower tail-position `loop`/`recur` with scalar bindings to a Janet - `while` + `var`/`set` instead of a recursive closure. Closes the 1.43×, taking - `mandelbrot` from 15.4× → ~10.8× JVM. Filed separately (see epic children). - -## Open questions answered - -- **Are Janet numbers boxed?** No — already unboxed. The `while` leg does pure - double arithmetic at a steady 153 ms with no allocation and no GC stutter, and - matches the other legs bit-for-bit. Janet's `number` is an immediate IEEE - double (stored inline in the Janet value, not heap-allocated). **Unboxing is - not a lever; it's done.** -- **GC share of `binary-trees`** — not measured here (the dev machine swaps - heavily, which distorts alloc-heavy benches; the handoff flags this). Size - lever 2 on a clean machine. The `mandelbrot` legs are alloc-free so are - unaffected and trustworthy. -- **Janet native-module / incremental C path** — not yet confirmed; this is the - gating question for the lever-1 spike (hot fns → C, rest → bytecode). - -## Artifacts (kept in `bench/`) - -- `mandelbrot-hand.janet` — optimal `while` Janet (the Janet VM floor reference) -- `mandelbrot-hand-rec.janet` — recursive-closure Janet (the loop-lowering diagnostic) -- `dump-mandelbrot-emit.janet` — dumps the Janet jolt emits for the hot fns - -The bench harness (`bench/run.sh`) ignores these (it iterates a fixed bench list). diff --git a/docs/libraries.md b/docs/libraries.md index 8b03f95..046918e 100644 --- a/docs/libraries.md +++ b/docs/libraries.md @@ -1,7 +1,7 @@ # Clojure libraries known to work with Jolt Libraries confirmed to load and pass their conformance checks on Jolt -(see `test/integration/deps-conformance-test.janet` and the +(see the [examples](https://github.com/jolt-lang/examples), e.g. the [ring-app example](https://github.com/jolt-lang/examples/tree/main/ring-app)). * [config](https://github.com/yogthos/config) @@ -19,7 +19,7 @@ Libraries confirmed to load and pass their conformance checks on Jolt * [honeysql](https://github.com/seancorfield/honeysql) — full formatter + helpers (select/insert/update/delete/joins/:inline), loaded unmodified from git * [clojure.jdbc](https://github.com/yogthos/clojure.jdbc) — as [jolt-lang/db](https://github.com/jolt-lang/db)'s - `jdbc.core`, reimplemented over janet sqlite3/pq drivers (SQLite + PostgreSQL) + `jdbc.core`, over the built-in SQLite access (libsqlite3 via Chez's FFI) * [next.jdbc](https://github.com/seancorfield/next-jdbc) — a compatibility layer in [jolt-lang/db](https://github.com/jolt-lang/db) (`next.jdbc`, `next.jdbc.sql`, `next.jdbc.prepare`, `next.jdbc.transaction`) over `jdbc.core`, for libraries @@ -33,10 +33,7 @@ Libraries confirmed to load and pass their conformance checks on Jolt `logf`/`logp`, `spy`, and `enabled?` all work; output goes to stderr. * [migratus](https://github.com/yogthos/migratus) — database migrations; loads unmodified and runs filesystem SQL/EDN migrations against SQLite through the - next.jdbc layer above. `migrate`/`rollback` round-trip end to end. Caveat: - migration ids are 14-digit timestamps, and the janet-lang/sqlite3 driver - currently truncates INTEGER columns to 32 bits, so completion tracking needs - the one-line upstream fix (`sqlite3_column_int64`); ids under 2^31 work as is. + next.jdbc layer above. `migrate`/`rollback` round-trip end to end. * [malli](https://github.com/metosin/malli) — data schema validation, on the [malli-app example](https://github.com/jolt-lang/examples/tree/main/malli-app). `m/validate` and `m/explain` work across the vocabulary (predicates, `:int`/ diff --git a/docs/rfc/0001-language-specification.md b/docs/rfc/0001-language-specification.md index 06222e7..bc078f9 100644 --- a/docs/rfc/0001-language-specification.md +++ b/docs/rfc/0001-language-specification.md @@ -170,9 +170,9 @@ Signature(s), since-version ## Open questions 1. Numerics: the reference has longs/doubles/ratios/BigInt with promotion - rules; CLJS has JS numbers; jolt has Janet numbers. Likely answer: specify - an integer/float core with a host-numeric-tower extension point — needs - its own design note in §4. + rules; CLJS has JS numbers. Resolved: jolt carries the Scheme numeric tower + (exact integers/bignums, exact ratios, flonum doubles), matching the + reference's tower — see the numerics note in §4. 2. Where do `*print-length*`-style dynamic vars land — host-dependent interface or portable with defaults? 3. License/venue if the spec outgrows this repo (likely CC-BY; separate repo diff --git a/docs/rfc/0003-transients.md b/docs/rfc/0003-transients.md index 8c451db..cee6dc5 100644 --- a/docs/rfc/0003-transients.md +++ b/docs/rfc/0003-transients.md @@ -1,50 +1,49 @@ -# RFC 0003: Transients — semantics and why they live in the Janet seed +# RFC 0003: Transients — semantics and the Chez mutable backing Status: accepted (design note) This note pins down what transients *are* in Jolt, where their behavior -deviates from JVM Clojure and why, and why the transient machinery is part of -the irreducible Janet seed rather than a candidate for the core-in-Clojure -migration (jolt-tzo). It exists so the kernel-shrink ladder doesn't revisit +deviates from JVM Clojure and why, and how the transient machinery is +represented in the Chez runtime. It exists so the design doesn't revisit transients every round. ## What a transient is in Jolt -A transient is a tagged Janet table wrapping a *native* mutable host value -(`core.janet`, "Transients" section): +A transient is a Chez record (`jolt-transient`, `host/chez/transients.ss`) +wrapping *true mutable* host backing, snapshotted to the immutable collection on +`persistent!`. The backing is per kind: -- transient vector — `@{:jolt/type :jolt/transient :kind :vector :arr ARRAY}`, - a Janet array. -- transient map — `:kind :map :tbl TABLE`, a Janet table mapping - `canon-key(k)` → `@[k v]`. Keying by canonical key keeps collection keys - comparing by value across representations (`[1 2]` the pvec and `[1 2]` the - tuple are one key), and storing the `@[k v]` pair preserves the *original* - key for the rebuilt persistent map. -- transient set — `:kind :set :tbl TABLE` mapping `canon-key(x)` → `x`. +- transient vector — a growable Scheme vector (a capacity buffer plus a fill + count `n`). `conj!`/`pop!` are in-place, amortized O(1); the buffer doubles on + growth. +- transient map — a Chez hashtable keyed by `key-hash` / `jolt=` + (value-equality, nil-safe). Hashing by value keeps collection keys comparing + across representations. +- transient set — a Chez hashtable of elements. +- `cow` — a copy-on-write fallback for anything else (e.g. a sorted coll). -`transient` accepts pvecs, mutable-build arrays, tuples (reader vectors and -map entries — added in the seed-shrink rounds so `(into [] (first {:a 1}))` -works through the vector fast path), sets, phms, and untagged struct maps. -Sorted collections are rejected, as on the JVM (not editable). +`transient` accepts pvecs, pmaps, psets, and the exotic colls handled by the +`cow` path. Each kind copies its source into the matching mutable backing once. -The bang ops (`conj!`, `assoc!`, `dissoc!`, `disj!`, `pop!`) mutate that host -value in place and return the transient — O(1) per op (amortized for array -push). `persistent!` rebuilds a persistent value from the host value and -invalidates the transient (`:jolt/persistent` flag; any further bang op or a -second `persistent!` throws "Transient used after persistent! call", matching -Clojure's invalidation contract). +The bang ops (`conj!`, `assoc!`, `dissoc!`, `disj!`, `pop!`) mutate that backing +in place and return the transient — O(1) per op (amortized for the vector push). +`persistent!` snapshots a persistent value from the backing (folding the +hashtable into a pmap/pset, handing off the buffer as a pvec) and invalidates the +transient (the record's active flag clears; any further bang op or a second +`persistent!` throws "transient used after persistent!", matching Clojure's +invalidation contract). Read ops work on an active transient where Clojure supports them: `get`, -`contains?`, `count`, and `nth` (vector kind) branch on the transient tag. +`contains?`, `count`, and `nth` (vector kind) see through the transient. `seq` on a transient is not supported, as in Clojure. ## Deviations from JVM Clojure (deliberate) **O(n) edges, O(1) middle.** Clojure's `(transient v)` is O(1) — the transient *shares* the persistent trie and marks nodes editable; `persistent!` is O(1) -too. Jolt's `transient` copies the source into a native array/table (O(n)) and -`persistent!` rebuilds (O(n)). The bang ops in between are native-host O(1), -which is *faster* per-op than trie editing. So the asymptotics of the usual +too. Jolt's `transient` copies the source into a mutable buffer/hashtable (O(n)) +and `persistent!` snapshots back (O(n)). The bang ops in between are host-mutable +O(1), which is *faster* per-op than trie editing. So the asymptotics of the usual pattern (persistent! (reduce conj! (transient []) coll)) @@ -53,13 +52,12 @@ are identical (O(n) total either way) with a better constant in the loop and a worse constant at the two edges. The pattern transients exist for — batch construction — is fully served. What is NOT served is transient-editing a *large* collection to change a few keys: that's O(n) in Jolt vs O(log n) in -Clojure, because `transient` flattens the pvec trie / phm HAMT into a -native array/table and `persistent!` rebuilds them. +Clojure, because `transient` copies the source into a growable Scheme vector / +Chez hashtable and `persistent!` snapshots it back. **No thread-ownership check.** JVM Clojure ≥1.7 also dropped the owner-thread assertion (for fork/join), keeping only "don't use after persistent!", which -Jolt enforces. Jolt code is fiber-concurrent; when real OS-thread futures land -(jolt-ejx), a transient handed across threads is a data race exactly as in +Jolt enforces. A transient handed across threads is a data race exactly as in Clojure — documented, not checked, same as the JVM. **`(conj!)` / `(conj! t)` arities** follow Clojure's transducer-era contract: @@ -68,51 +66,43 @@ zero args makes a fresh `(transient [])`, one arg returns it untouched. lenient kvs walk of Jolt's `assoc`. **No transient sorted variants** — same as Clojure. One leniency: Clojure -throws on `(transient '(1))`, but Jolt's lists are Janet arrays underneath and -fall into the mutable-build branch, yielding a transient *vector*. Harmless -(the result of `persistent!` is a vector, never silently a list) but -non-Clojure; tighten if it ever bites. +throws on `(transient '(1))`, but Jolt routes a list through the `cow` fallback +path, yielding a transient. Harmless but non-Clojure; tighten if it ever +bites. -## Why transients stay in the Janet seed +## Why transients live in the host -The migration ladder (jolt-tzo) moves anything expressible as *pure Clojure -over existing primitives* out of the seed. Transients fail that test on three +Transients are part of the value/representation layer in the Chez runtime +(`host/chez/transients.ss`), not the portable `clojure.core` overlay, on three grounds: 1. **They are the mutation kernel.** A transient's entire value is direct - mutation of a host array/table. The overlay's only mutation seam is - `jolt.host/ref-put!` (a single table-put). Re-expressing `tr-conj!` etc. in - Clojure would mean either growing the host surface one-for-one - (`host-array-push!`, `host-table-put!`, …, i.e. moving the same code behind - more indirection) or simulating mutation over persistent values (defeating - the point of transients). Either way the Janet line count moves, it doesn't - shrink. + mutation of a host buffer/hashtable. The overlay has no mutation seam of its + own. Re-expressing the bang ops in Clojure would mean either growing the host + surface one-for-one (a host-vector-push, a host-hashtable-put, …, i.e. moving + the same code behind more indirection) or simulating mutation over persistent + values (defeating the point of transients). -2. **They sit under the seed's own dispatch.** `conj`/`assoc`/`get`/`count`/ - `contains?` in the seed branch on the transient tag. Hoisting the transient - ops above that dispatch (the hierarchy-port pattern of lazily-resolved - overlay vars) would put an interpreted/compiled-Clojure call inside the - hottest native paths for no semantic gain — transients have no semantics to - *fix* (unlike hierarchy, which had real correctness gaps). +2. **They sit under the collection dispatch.** `conj`/`assoc`/`get`/`count`/ + `contains?` see through a transient. Hoisting the transient ops above that + dispatch would put a compiled-Clojure call inside the hottest paths for no + semantic gain — transients have no semantics to *fix*. -3. **The value layer is declared irreducible.** The self-hosting design doc - (docs/self-hosting-compiler.md, "The kernel") keeps the value/representation - layer — persistent collections and, with them, their mutable scratch - counterparts — in the host. Transients are representation, not library. +3. **The value layer is the host's job.** The persistent collections and, with + them, their mutable scratch counterparts, live in the Chez runtime alongside + the value model. Transients are representation, not library. -What CAN move (and mostly has): anything *derived* — e.g. `into`'s -transient-using fast path, or future `update!`-style conveniences — is plain -Clojure over `transient`/bang-ops/`persistent!` and belongs in the overlay -tiers as ordinary migration batches. +What lives in the overlay: anything *derived* — e.g. `into`'s transient-using +fast path, or `update!`-style conveniences — is plain Clojure over +`transient`/bang-ops/`persistent!`. ## Future work -- pvec is already a 32-way trie with structural sharing (pv.janet), so - Clojure-style O(1) `transient`/`persistent!` via editable nodes is a real - option for vectors — an internal change behind the same surface, not a - semantics change. phm is now a HAMT with structural sharing too (jolt-684u), - and sorted maps/sets are a red-black tree (jolt-0hbr), so the same editable- - node trick is open for those as well — the transient surface here is still the - copy-to-native-table flatten. +- The persistent map/set are a bitmap HAMT with structural sharing + (`host/chez/collections.ss`), so Clojure-style O(1) `transient`/`persistent!` + via editable nodes is a real option there — an internal change behind the same + surface, not a semantics change. The persistent vector is a flat + copy-on-write Scheme vector rather than a trie, so the transient surface for + it stays the copy-to-growable-vector path. - `transient?` (Jolt extension, useful in tests) stays; Clojure has no public predicate, so it must not leak into portability-sensitive code. diff --git a/docs/rfc/0004-type-hints.md b/docs/rfc/0004-type-hints.md index 5c47070..4fa2e3e 100644 --- a/docs/rfc/0004-type-hints.md +++ b/docs/rfc/0004-type-hints.md @@ -11,26 +11,19 @@ measured effect, so later work does not relitigate it. ## Background: why the lookup carries a guard A Jolt map value has several runtime representations (see RFC on collections and -`src/jolt/core.janet`): a Janet struct for a small all-scalar-key literal map, a -persistent hash map (a table tagged `:jolt/type :jolt/phm`) when a key is a -collection or a value is nil, plus sorted maps, transients, and record/deftype -instances. A record instance is a Janet table tagged `:jolt/deftype` but, like a -struct, it carries no `:jolt/type`, so a raw Janet `(get inst :field)` reads its -fields directly. +`host/chez/collections.ss`): a persistent hash map (a bitmap HAMT) for the +general case, plus sorted maps, transients, and record/deftype instances. A +record instance is a Chez record (`jrec`) whose fields are read directly off the +record's storage, while a HAMT lookup runs the full `jolt=`/`jolt-hash`-keyed +collection path. -A constant-keyword lookup `(:k m)` compiles to a guarded form: - -```janet -(if (get m :jolt/type) (core-get m k) (get m k)) -``` - -The guard is one opcode. A non-nil `:jolt/type` routes phm/sorted/transient/ -lazy-seq values to `core-get`'s full semantics; everything else (structs, -records, nil, scalars) takes the bare Janet `get`, which matches `core-get` for -keyword keys. The guard is correct and cheap, but on a struct it is a second -`get`: profiling the ray tracer (a naive all-maps program) found keyword lookups -are about half of a render, and the guard is the only avoidable part of each -one. A bare get is roughly 20ns where the guarded form is roughly 36ns. +A constant-keyword lookup `(:k m)` compiles to a guarded form: it inspects the +subject's representation and routes a HAMT/sorted/transient/lazy-seq value to the +full `jolt-get` semantics, while a record/raw-get-safe value takes the direct +field read, which matches `jolt-get` for keyword keys. The guard is correct and +cheap, but on a raw-get-safe value it is wasted work: profiling the ray tracer (a +naive all-maps program) found keyword lookups are about half of a render, and the +guard is the only avoidable part of each one. Dropping the guard is only safe when the subject is known to be a plain struct/record rather than a tagged collection. Jolt does not infer that @@ -59,27 +52,27 @@ optimization. ## How it flows The reader already keeps `^hint` metadata on the binding symbol and is otherwise -transparent (`reader.janet`, `meta-form->map`). The change threads that fact to -the lookup site: +transparent (`host/chez/reader.ss`). The change threads that fact to the lookup +site: 1. The analyzer (`jolt-core/jolt/analyzer.clj`) records a `:struct` hint per local in its env when a param or `let` binding carries `^:struct` or a record-type tag, and attaches `:hint :struct` to that local's `:local` IR - node. Resolving a record-type tag uses a new host contract function - `record-type?` (`src/jolt/host_iface.janet`), which checks for the `->Name` - constructor. -2. The back end (`emit-kw-lookup` in `src/jolt/backend.janet`) emits the bare get + node. Resolving a record-type tag uses the host contract function + `record-type?` (`jolt.host`, backed by `host/chez/host-contract.ss`), which + checks for the `->Name` constructor. +2. The back end (`jolt-core/jolt/backend_scheme.clj`) emits the direct field read when the lookup subject is a `:local` carrying the hint, and the guarded form - otherwise. The unhinted path is byte-identical to before. + otherwise. The unhinted path is identical to before. 3. The inline pass (`jolt-core/jolt/passes.clj`) propagates the hint: when it binds a non-trivial call argument to a fresh local, it carries the called function's parameter hint onto that local, so lookups inside the spliced body - keep the bare path. Without this, inlining a hinted function would erase the + keep the direct path. Without this, inlining a hinted function would erase the benefit, because the hinted parameter is replaced by an unhinted temporary. The same machinery covers both `(:k m)` and `(get m :k [default])` when the key is a constant keyword. A `get` with a variable, numeric, or string key falls -through to `core-get` unchanged. +through to `jolt-get` unchanged. ## Record hints across namespaces, and as inference seeds @@ -98,15 +91,14 @@ the function where the hot reads actually happen. **It resolves across namespaces.** A hint may name a record defined in another namespace, in either spelling — `^Vec3` where the type is `:refer`-ed, or -`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key` in -`src/jolt/host_iface.janet`, backed by `record-hint-ctor-key` in -`src/jolt/evaluator.janet`) runs against the *compile* namespace and maps the -type to its home constructor key through a constructor-value index — keyed by the -constructor value, not a var's namespace, so a `:refer`-interned var (whose -namespace is the referring one) still resolves home. The reader keeps a tag's -namespace qualifier (`^v/Vec3` → `"v/Vec3"`, not `"Vec3"`) so the aliased -spelling has something to resolve. Both `defrecord` field hints and function -parameter hints use this resolution. +`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key`, +a `jolt.host` contract function backed by `host/chez/host-contract.ss`) runs +against the *compile* namespace and maps the type to its home constructor key +through a constructor-value index — keyed by the constructor value, not a var's +namespace, so a `:refer`-interned var (whose namespace is the referring one) +still resolves home. The reader keeps a tag's namespace qualifier (`^v/Vec3` → +`"v/Vec3"`, not `"Vec3"`) so the aliased spelling has something to resolve. Both +`defrecord` field hints and function parameter hints use this resolution. ## Soundness and the checked mode @@ -120,8 +112,8 @@ To make a lie visible without taxing the fast path, `JOLT_CHECK_HINTS=1` keeps the guard but throws on the tagged arm with a message naming the local and key: ``` -type hint violated on `m`: (:a m) — value carries :jolt/type -(a phm/sorted/transient/lazy-seq), not the plain struct/record the +type hint violated on `m`: (:a m) — value is a +phm/sorted/transient/lazy-seq, not the plain struct/record the ^:struct/^Record hint asserts ``` @@ -134,7 +126,7 @@ off). The flag is part of the image-cache fingerprint. Type hints parse in every position Clojure accepts them and are inert except for the optimization above. This matches Clojure's "parse and otherwise do nothing" model, with the difference that Clojure additionally uses hints to avoid -reflection and select primitive arithmetic, which do not apply to a Janet host. +reflection and select primitive arithmetic, which do not apply to the Chez host. ## Measured effect diff --git a/docs/rfc/0005-structural-type-inference.md b/docs/rfc/0005-structural-type-inference.md index 42cad41..f3e309b 100644 --- a/docs/rfc/0005-structural-type-inference.md +++ b/docs/rfc/0005-structural-type-inference.md @@ -50,7 +50,7 @@ A type `T` is one of: `:nonnil` for "provably not nil and not false", which is what the struct-vs-phm decision needs; see below.) - `:nil`. -- `{:struct {field -> T}}` — a raw-get-safe map (Janet struct or record) whose +- `{:struct {field -> T}}` — a raw-get-safe map (a record) whose field `k` has type `(fields k)` or `:any` if absent. The degenerate `{:struct {}}` is "a struct, fields unknown" and replaces today's `:struct-map`. @@ -65,7 +65,7 @@ A type `T` is one of: Types are immutable values comparable by structural equality, exactly like the current `{:vec ELEM}` representation, so they flow across the portable -inference and the Janet orchestrator unchanged. +inference and the host unchanged. ### Join (least upper bound) diff --git a/docs/seed-overlay-registry.md b/docs/seed-overlay-registry.md index 5bbcbcf..674a6b7 100644 --- a/docs/seed-overlay-registry.md +++ b/docs/seed-overlay-registry.md @@ -1,80 +1,45 @@ # Seed ↔ Overlay Registry -Jolt is "Clojure on Janet": a shrinking **Janet seed** (`src/jolt/*.janet`) -hosts a **Clojure overlay** (`jolt-core/clojure/core/NN-*.clj`). Both define -`clojure.core`-facing functions, and for a handful of names *both* tiers carry a -definition. Which copy is authoritative has been tribal knowledge. This document -is the single source of truth; `test/unit/seed-overlay-registry-test.janet` is a -build-time drift check that fails if reality diverges from what is written here. +Jolt is Clojure on Chez Scheme. `clojure.core` is built from two tiers that both +define `clojure.core`-facing vars, and for a handful of names *both* tiers carry +a definition. This document records how the two tiers relate and which copy is +authoritative. -## The registration mechanism +## The two tiers -Seed core functions are named with a `core-` prefix (`core-into`, `core-conj`, -`core-transduce`) and registered into the `clojure.core` namespace by the -`core-bindings` table in `src/jolt/core.janet`. Each entry maps a **public -Clojure name** (the string key) to a seed function value: +- **Native shims** (`host/chez/natives-*.ss`) bind a set of `clojure.core` vars + directly to Scheme runtime values via `def-var!` — collection constructors, + seq fns, numeric/string ops, and so on. These cover names the overlay assumes + exist as bare `clojure.core` vars but does not define itself. +- **The Clojure overlay** (`jolt-core/clojure/core/NN-*.clj`) defines the rest of + `clojure.core` in dependency-ordered tiers, loaded in order: `00-syntax`, + `00-kernel`, `10-seq`, `20-coll`, `25-sorted`, `30-macros`, `40-lazy`, `50-io`. -```janet -(def- core-bindings - @{"into" core-into - "reduce" core-reduce - ...}) -``` - -`init-core!` (`src/jolt/core.janet`) interns every pair into `clojure.core`. -The overlay tiers load afterwards (`api.janet`: 00-syntax, 00-kernel, 10-seq, -20-coll, 25-sorted, 30-macros, 40-lazy, 50-io). When an overlay tier `(defn X …)` -for a name that `core-bindings` already registered, the **overlay def shadows the -seed binding** — the seed `core-X` then survives only if some other seed code -still calls it directly. +The overlay loads after the native shims. When an overlay tier `(defn X …)` for a +name a native shim already bound, the **overlay def shadows the native binding** — +user code sees the overlay copy. The native binding then survives only if some +other native/runtime code still calls the Scheme value directly. So a name's *home* is determined by two facts: -1. is it a key in `core-bindings`? (registered ⇒ the seed `core-X` is reachable) -2. does an overlay tier `(defn X …)`? (defined ⇒ the overlay copy shadows) +1. is it bound by a native shim? (the Scheme value is reachable from the runtime) +2. does an overlay tier `(defn X …)`? (the overlay copy is what user code sees) -## Dispatch-only seed helpers: the `__` prefix +## The compiled seed -Seed functions that are **not** public Clojure vars but must be reachable by -name from compiled/overlay code (compiler hooks, macro-expansion targets) are -registered under a `__`-prefixed key — e.g. `"__sq1"`, `"__write"`, -`"__bit-and"`, `"__jdbc-conn-raw"`. The `__` prefix is unreadable as a -user-level symbol, so these never collide with or masquerade as public API. When -you add a dispatch-only hook, give it a `__` key; do not register it under a bare -name. +`clojure.core` is compiled ahead of time into the checked-in seed +(`host/chez/seed/{prelude,image}.ss`) as Scheme `def-var!` forms. The seed's +source twin is the overlay (`jolt-core/clojure/core/*.clj` plus the stdlib +namespaces under `src/jolt/clojure/`); `host/chez/emit-image.ss` re-emits the +prelude from those sources on Chez. The build is a byte-fixpoint: rebuilding from +an up-to-date seed reproduces it exactly. -## Dispatch twins +## Consistency guard -A **twin** is a name with *both* a seed `core-X` defn and an overlay `(defn X …)`. -There are exactly five. Each seed site carries a greppable `SEED-TWIN:` comment. - -| name | overlay (authoritative public) | seed copy (`core-X`) | registered? | role of the seed copy | -|---------------|----------------------------------------|--------------------------------------|-------------|------------------------| -| `char?` | `20-coll.clj` `char?` | `core_types.janet` `core-char?` | no | internal type dispatch | -| `sorted-map?` | `25-sorted.clj` `sorted-map?` | `core_types.janet` `core-sorted-map?`| no | internal dispatch (sorted-op) | -| `sorted-set?` | `25-sorted.clj` `sorted-set?` | `core_types.janet` `core-sorted-set?`| no | internal dispatch | -| `sorted?` | `25-sorted.clj` `sorted?` | `core_types.janet` `core-sorted?` | no | internal dispatch | -| `transduce` | `20-coll.clj` `transduce` | `core_coll.janet` `core-transduce` | no | internal helper for `core-into` only | - -None of the five is registered in `core-bindings`: the overlay copy is the public -one, and the seed copy is reached only by other *seed* code (so editing the seed -copy alone will not change what user code sees — change both, or move the logic). - -## The surprising asymmetry: `into` vs `transduce` - -`into` and `reduce` are **seed-public**: registered in `core-bindings`, and the -overlay deliberately does *not* redefine them (they sit on the perf wall — see -the "into stays in the seed" note in `20-coll.clj`). `transduce`, by contrast, is -**overlay-public**: the overlay `transduce` is the real one, and `core-transduce` -remains only because `core-into` calls it directly. So two functions that read as -a matched pair have opposite homes. That asymmetry is intentional and is the -reason this registry exists. - -## Drift check - -`test/unit/seed-overlay-registry-test.janet` recomputes the twin set from source -(names with both a seed `core-X` defn and an overlay `defn X`) and asserts it -equals the five above. It also asserts none of the twins is registered in -`core-bindings`, and that every non-`__` `core-bindings` key is a plausible -public name (no accidental `__`-less dispatch helper). If you add, remove, or -re-home a twin, update this table and that test together. +There is no separate drift-check test for the registry. The self-hosting +fixpoint is the guard: after changing a seed source (a core tier, the compiler +namespaces, the host contract, the reader, or `emit-image.ss`) you must re-mint +the seed (`make remint`), and `make selfhost` fails if the checked-in seed and +its sources have drifted. So if the overlay's shadowing relationship changes, the +re-minted prelude changes with it, and the fixpoint check keeps source and seed +in agreement. diff --git a/docs/self-hosting-architecture.md b/docs/self-hosting-architecture.md deleted file mode 100644 index e208b87..0000000 --- a/docs/self-hosting-architecture.md +++ /dev/null @@ -1,138 +0,0 @@ -# Self-hosting architecture: portable jolt-core over a host runtime - -Design for splitting Jolt into a **portable Clojure-in-Clojure core** and a -**host runtime** (Janet today, another runtime tomorrow), so the language is -truly self-hosted and `jolt-core` can be lifted out and re-hosted. - -This is the design that must be right *before* writing the compiler in Clojure — -see [[self-hosting-compiler]] for the staged plan it plugs into. - -## What "truly self-hosted + portable" requires - -Two independent properties: - -1. **Self-hosted** — the compiler and most of `clojure.core` are written in - Clojure and compiled by Jolt itself. -2. **Portable** — that Clojure code (`jolt-core`) depends only on a small, - explicit **host contract**, never on Janet directly. Re-hosting means - implementing the contract for a new runtime; `jolt-core` is reused verbatim. - -The enemy is `jolt-core` calling `janet/tuple`, `make-vec`, `ns-find`, etc. -directly — that welds it to Janet. Every host dependency must go through the -contract. - -## Prior art (the seam everyone uses) - -- **Clojure (JVM).** `clojure.lang.*` (Java) is the host: `RT`/`Numbers` runtime - helpers, the `Compiler` (form → JVM bytecode), persistent data structures, - `Var`/`Namespace`, the reader. `clojure/core.clj` is the language, in Clojure. - Seam: ~20 primitive special forms + `RT` static methods. Everything else is - Clojure. -- **ClojureScript (self-hosted).** Two portable passes — `cljs.analyzer` - (form → AST **as data**, reading a **compiler-state map** of - namespaces/defs/macros, *not* host objects) and `cljs.compiler` (AST → JS, the - host-specific back end). `cljs.core` is Clojure compiled to JS. Platform splits - live in `.cljc` reader conditionals. This is the closest model to what we want: - **the analyzer is host-agnostic; only the back end and the runtime are - host-specific.** -- **Nanopass / Guile Tree-IL.** A high-level IR is the portability seam; multiple - back ends consume it. -- **ClojureCLR / ClojureDart / jank.** Same shape every time: portable analyzer + - host back end + host runtime. - -The invariant across all of them: **the IR (analyzer output) and a small runtime -protocol are the contract; the front end is portable, the back end and runtime -are per-host.** - -## Decisions (locked) - -- **Seam = a minimal host protocol.** `jolt-core` calls a small documented set of - host fns (in ns `jolt.host`): `resolve-sym`, `macro?`, `macroexpand-1`, - `current-ns`, `intern!`, plus the `RT` primitives. Each host provides `jolt.host` - (+ RT). Re-hosting = reimplement that handful of fns. The protocol *is* the - boundary; `jolt-core` never touches Janet directly. -- **Physical split now.** Portable Clojure lives under `jolt-core/` (a new source - root, embedded into the binary like the rest of the stdlib); host Janet code for - the new pipeline under `host/janet/`. Legacy host modules under `src/jolt/*.janet` - are the existing Janet host and get relocated under `host/janet/` in a later - mechanical pass (tracked) — not moved big-bang now, to keep the suite green. - -## The Jolt split - -``` -jolt-core/ PORTABLE Clojure — no Janet. Depends only on the contract. - ir the IR spec (data shapes the analyzer emits) - analyzer form -> IR (macroexpands; resolves via host protocol) - macros when/cond/->/defn/... (the macro library, in Clojure) - core clojure.core fns expressible in Clojure, over RT primitives - -host/janet/ THE HOST — Janet. Implements the contract. - reader text -> jolt forms - rt data structures + RT primitive fns (cons/first/+/get/apply…) - backend IR -> Janet forms -> Janet compile -> bytecode (the emitter) - cenv the compile-time host protocol impl (resolve/macro?/intern) - bootstrap load jolt-core, wire analyzer+backend into the loader - interop janet.* bridge -``` - -Two contracts cross the seam: - -### 1. The IR (analyzer → back end) -The existing `:op`-tagged AST, made **host-neutral**: -- `{:op :const :val v}`, `:if`, `:do`, `:let`, `:fn` (arities), `:invoke`, - `:vector`/`:map`/`:set`, `:quote`, `:throw`/`:try`, `:loop`/`:recur`. -- **Globals reference vars by NAME, not by host cell:** - `{:op :var :ns "clojure.core" :name "map"}`. (compiler.janet today embeds the - Janet var cell as a constant — that's a host leak and breaks AOT. Name-based - refs are both portable and AOT-friendly; the back end resolves the cell.) -- No embedded host function values. Calls to runtime primitives are - `{:op :rt :name "cons"}` resolved by the back end to the host's RT fn. - -### 2. The host contract (two protocols) -- **Compile-time (`cenv`)** — what the analyzer needs from the host while - analyzing: `(current-ns)`, `(resolve-sym sym) -> {:kind :var|:macro|:local|:special|:host, :ns, :name}`, - `(macroexpand-1 form)`, `(intern! ns sym meta)`. The analyzer calls only these; - it never touches Janet ns/var tables. (CLJS keeps this as pure data; we use a - small protocol — a minimal, documented boundary — because Jolt already has live - ns/var objects. The protocol *is* the seam.) -- **Runtime (`RT`)** — the primitive fns emitted code and `jolt-core` call by - stable name: arithmetic/compare, `cons/first/rest/seq/conj/get/assoc/count`, - `apply`, `=`, vector/map/set constructors, var deref/bind, keyword/symbol - construction. The back end maps each to the host (on Janet, mostly the existing - `core-*`). To re-host, implement this set. - -## Why name-based vars (not embedded cells) - -`compiler.janet` compiles a global ref to a closure over the Janet var cell. That -(a) is a Janet value baked into the IR — not portable, and (b) can't be marshaled -for AOT without the runtime-dict trick. Compiling instead to *resolve var by -(ns,name) at call time* through an RT primitive keeps redefinition live, makes the -IR host-neutral, and makes images trivially portable. The per-call lookup is the -cost; it can be cached/direct-linked later as an opt-in optimization. - -## Bootstrap & staging (keeps the suite green throughout) - -`compiler.janet` stays as the **bootstrap back end** until the Clojure pipeline is -proven. Order: - -1. **Freeze the IR** spec and refactor `compiler.janet`'s emit to consume - name-based `:var` (no behavior change; bootstrap still works). -2. **Define the host contract** (`cenv` + `RT`) and implement it on Janet, - exposed under a stable namespace the Clojure core can call. -3. **Write `jolt.analyzer` in Clojure** producing IR, against `cenv`. Diff its IR - against the Janet analyzer on the conformance corpus until identical. -4. **Janet back end consumes IR** from the Clojure analyzer; wire into the loader - behind a flag. Validate at parity (dual-mode conformance + clojure-test-suite). -5. **Flip** the loader to the Clojure analyzer + Janet back end; `compiler.janet` - shrinks to the back end only. -6. **Move `clojure.core`** macros then fns into `jolt-core` incrementally, each - compiled by the prior stage, isolating host bits behind `RT`. - -Guards at every step: the dual-mode conformance harness (interpret vs compile) -and the clojure-test-suite baseline. - -## The portability test - -When done, re-hosting Jolt to runtime X means writing only: `host/X/{reader, rt, -backend, cenv, bootstrap}`. `jolt-core/{ir, analyzer, macros, core}` is reused -unchanged. That is the concrete bar for "truly self-hosted and portable." diff --git a/docs/self-hosting-compiler.md b/docs/self-hosting-compiler.md deleted file mode 100644 index b46b57e..0000000 --- a/docs/self-hosting-compiler.md +++ /dev/null @@ -1,175 +0,0 @@ -# Toward a self-hosting Jolt compiler - -Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc -compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet -bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet -bootstrap. This is a design doc, not a changelog — it describes where we are, the -prior art, the constraints we verified, and a recommended path. - -## The goal - -- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is - enough to start; the rest of Clojure — including the compiler — is written in - Clojure and compiled by Jolt itself, growing the language as it compiles more - of itself. -- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast), - not tree-walking. -- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols, - multimethods, and everything else stay live and redefinable at the REPL even - for compiled code. -- **Minimal host requirement.** Shrink what must exist in Janet to the - irreducible base. - -## Where Jolt is today - -- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a - tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the - stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in - the host, inverted from the Clojure-in-Clojure ideal. -- The interpreter (`eval-form`) is the complete reference path. -- The compiler (`compiler.janet`) — `analyze-form` (reader form → `:op` AST) → - `emit` (AST → Janet form) → Janet `compile`/`eval` — is now **on by default** - in the shipped runtime (`JOLT_INTERPRET=1` opts out). It is a *hybrid*: forms - it can't compile correctly throw `jolt/uncompilable` and fall back to the - interpreter (`loader/eval-toplevel`), so results always match the interpreter. - Validated at parity — conformance 218/218 under both interpret and compile, and - the clojure-test-suite under compile passes 3932 (vs the 3913 interpreter - baseline) across ~4.6k assertions. -- Done so far: var-indirection (globals deref through var cells, so compiled code - is REPL-redefinable); hybrid fallback; compilation of multi-arity / named / - variadic fns and `recur` inside `fn`; map and vector literal compilation - (mode-correct via `make-vec` / `build-map-literal`); resolution that mirrors - the interpreter (current ns → `clojure.core` → Janet-env fallback); and AOT - (`aot.janet`) that marshals a compiled namespace to a Janet bytecode image - against the baked-in runtime dictionary and loads it back. -- Still open — the actual self-hosting: the compiler and most of `clojure.core` - are still Janet. Rewriting them in Clojure (compiled by Jolt) is the remaining - Clojure-in-Clojure work. - -## What the host gives us (verified) - -Janet already is the backend and the AOT story — we don't need a custom bytecode -emitter: - -- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is - Clojure form → correct Janet form → `compile`. -- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled - environment to a **bytecode image** and load it back: this is Phase 4 AOT. -- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the - form layer (we shouldn't need to). - -**The catch we verified:** Janet *early-binds* top-level references. Compile -`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns -the old value. So emitting Jolt globals as plain Janet symbols (what the current -compiler largely does) is fundamentally incompatible with REPL redefinition. This -is the single most important design constraint below. - -## Prior art - -- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is - written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only - ~20 special forms are primitive; everything else is macros/functions. Crucially, - compiled call sites go **through Var objects** (a deref), so redefining a var is - visible to existing compiled callers — that's how speed and live redefinition - coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the - var indirection) for speed where you don't need redefinition (used for core in - production). AOT compiles namespaces to `.class` files. -- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus - a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS). - `cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at - ~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we - want with Janet as the backend. -- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally - specified* intermediate languages, with autogenerated boilerplate to recur - through unchanged forms and checks that each pass's output matches its grammar. - The lesson for "grow the language as it compiles itself": keep passes small and - IRs explicit so adding a form is local and verifiable. -- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS - (optimization IR) → VM bytecode, with several front-end languages targeting - Tree-IL. The closest analog to "Lisp → bytecode on a VM." - -## Assessment: is the current approach the right one? - -The overall *shape* is right and matches ClojureScript: front-end (analyze → -emit) with the host VM as the backend, emitting host forms that the host compiles -to bytecode. Two things need to change to reach the goal: - -1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref - through the var cell**, not as a Janet symbol. Jolt vars are already cells - (`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly - `((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition - updates the cell's root, so compiled callers see it — exactly Clojure's model. - One indirection per global call; locals and control flow stay direct and fast. - Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition. -2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting - means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is - Clojure. That's the bulk of the work and where the "language builds itself" - payoff lives. - -So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for -free), fix global binding, and progressively self-host. - -## Recommended architecture - -**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small, -named passes rather than one big walker: - -1. *read* — reader → forms (already have it). -2. *macroexpand* — fully expand to special forms + calls (the interpreter already - expands; share one expander). -3. *analyze* — forms → AST, resolving locals vs vars and tagging ops. -4. *(optional) optimize* — constant-fold, direct-link hot calls, etc. -5. *emit* — AST → Janet form, with globals as var-cell derefs. -6. *compile* — Janet `compile` → bytecode; `make-image` for AOT. - -Make each pass total over the IR so an unhandled node is an explicit gap, not a -silent miss. - -**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in -the host before any Clojure can run: the reader; the value/representation layer -(vars, namespaces, symbols, keywords, persistent collections, chars); host -interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur` -evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`, -the macros, and the compiler — is Clojure loaded and (eventually) compiled by the -kernel. Today the kernel is far larger than this; shrinking it is a long game. - -**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass -can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`) -for that sub-form instead of erroring. This lets the compiler be incomplete and -still correct (hot paths compile, cold/unsupported paths interpret), lets us grow -coverage incrementally, and de-risks the self-hosting bootstrap. - -**Live flexibility.** Vars stay first-class cells; compiled code derefs them; -`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct -linking seals a call against redefinition, so the interactive modes — the REPL, -`-e`, the nREPL server — always stay live (indirect). Running a *program* (a -file, `-m`/`-M`) direct-links by default, since it's a closed world; opt back out -with `JOLT_NO_DIRECT_LINK`. (See RFC 0005, "Compilation modes and defaults".) - -## A staged path - -1. **Var-indirection in the emitter** — *done*. Global refs compile as var-cell - derefs, so a compiled `defn` is redefinable at the REPL. -2. **Hybrid fallback + coverage** (`jolt-1bj`) — *done*. Forms the compiler can't - compile throw `jolt/uncompilable` and fall back to the interpreter, so compile - mode is always correct. Covered: multi-arity/named/variadic fns, `recur` in - `fn`, map/vector literals, and resolution matching the interpreter. - Destructuring compiles via the shared `destructure` expander: the `fn`/`let`/ - `loop`/`defn` macros desugar to plain-symbol `fn*`/`let*`/`loop*`, so it no - longer falls back — and the primitives reject patterns outright, matching - Clojure (`jolt-f79`). -5. **Compile-by-default + AOT** (`jolt-7j9`) — *done, done out of order*. Once the - hybrid path was validated at parity, compilation was flipped on by default and - AOT images (`aot.janet`) landed. Done before 3–4 because it's the runtime - payoff and only needed the hybrid path to be correct, not self-hosting. -3. **Self-host the compiler** (`jolt-lcn`) — *open*. Rewrite `compiler.janet` as - Clojure (`jolt.compiler`) that Jolt compiles. Now the compiler is part of the - language it compiles. -4. **Shrink the kernel / core-in-Clojure** (`jolt-uqi`) — *open*. Move - `clojure.core` from Janet to Clojure incrementally, each piece compiled by the - previous stage — the language building itself — leaving a minimal Janet kernel. - -What remains (3 and 4) is the actual Clojure-in-Clojure rewrite: the largest part -of the work and where the "language builds itself" payoff lives. The correctness -and runtime foundations it needs — redefinable compiled code, an always-correct -hybrid path, compile-by-default, and AOT — are now in place. diff --git a/docs/spec/00-front-matter.md b/docs/spec/00-front-matter.md index 63d1d54..1e974952 100644 --- a/docs/spec/00-front-matter.md +++ b/docs/spec/00-front-matter.md @@ -68,8 +68,9 @@ Behavioral questions are settled in this order: differential testing against the reference implementation → cross-dialect agreement in clojure-test-suite → ClojureDocs community examples (verified before inclusion) → reference source (for intent). Conformance tests live in this repository -(`test/integration/conformance-test.janet` runs each assertion through three -independent execution paths) and in the cross-dialect clojure-test-suite. +(the corpus `test/chez/corpus.edn`, run on Chez via `host/chez/run-corpus.ss` +and certified against reference JVM Clojure by `test/conformance/certify.clj`) +and in the cross-dialect clojure-test-suite. ## 5. Chapter plan diff --git a/docs/spec/README.md b/docs/spec/README.md index 7c710f2..69f2ec0 100644 --- a/docs/spec/README.md +++ b/docs/spec/README.md @@ -38,10 +38,13 @@ Of the 694 `clojure.core` vars in the ClojureDocs inventory: ## How this connects to the test suites -- `test/integration/conformance-test.janet` — 302 assertions, each run - through three independent execution paths (interpreter, bootstrap - compiler, self-hosted compiler) that must agree. Spec entries cite these. -- `test/spec/*.janet` — ~1,500 behavioral cases organized by topic. +- `test/chez/corpus.edn` — the host-neutral behavioral corpus, one row per + case (`{:suite :label :expected :actual}`). The Chez compiler evaluates each + case via `host/chez/run-corpus.ss` (run with `make corpus`), and + `test/conformance/certify.clj` certifies every `:expected` against reference + JVM Clojure (run with `make certify`). Spec entries cite these cases. +- `test/conformance/` — the certification tooling and classified divergences + (`certify.clj`, `known-divergences.edn`); see its `README.md` and `SPEC.md`. - `vendor/clojure-test-suite` — the cross-dialect suite (≥4081 assertions passing); dialect splits there are classification evidence. - jank's per-construct corpus (`~/src/jank/compiler+runtime/test/jank`) is diff --git a/docs/tools-deps.md b/docs/tools-deps.md index 12999a2..84461ca 100644 --- a/docs/tools-deps.md +++ b/docs/tools-deps.md @@ -9,91 +9,57 @@ Scope, decided up front: - **pure `clj`/`cljc`** — anything needing the JVM won't load or run; expected. - **no classpath abstraction** — `require` just needs to find a dep's namespaces; "the classpath" is an ordered list of source directories. -- **piggyback on jpm** — reuse jpm's git fetch + cache; don't write a package - manager. +- **own resolver, own reader** — `deps.edn` is read by jolt's own reader, and git + fetch/cache is a thin shell-out to `git`; no external package manager. - **deps-agnostic runtime core** — resolution is a CLI front-end concern, not a - runtime one. The `jolt` *runtime* knows nothing about deps.edn; it only reads - source roots from `JOLT_PATH`. The `jolt` *CLI* resolves a deps.edn into that - env var before running, in a module (`deps.janet`) that loads `jpm` lazily. - (This was a separate `jolt-deps` binary originally; it was folded into `jolt` - for a single-binary UX — the code boundary stayed, only the executable merged. - A back-compat `jolt-deps` shim still ships and forwards to `jolt`.) + runtime one. The runtime knows nothing about `deps.edn`; it only consumes a + list of source roots. The CLI resolves a `deps.edn` into those roots before + running. -## How jpm handles dependencies +## How resolution works -jpm's package code (`jpm/pm.janet`) splits into a fetch half and a build half, -and we use only the first: +`jolt.deps` (`jolt-core/jolt/deps.clj`) reads `deps.edn` (jolt's own reader +parses the EDN), then walks `:deps`: -- **`resolve-bundle`** normalizes a dep spec to `{:url :tag :type :shallow}`, - accepting `:url`/`:repo` + `:tag`/`:sha`/`:commit`/`:ref`. A deps.edn - `{:git/url … :git/sha …}` maps straight onto it. -- **`download-bundle url :git tag shallow`** clones into a content-addressed cache - (`/.cache/git__`) and returns the path — - `git init` + `remote add` + fetch + reset, plus submodules. No build step. -- **`bundle-install`** is the half we skip: it then runs `project.janet` build - rules, which a Clojure lib doesn't have. It's cleanly separable from the clone. - -So jpm gives us git resolution and a cache for free; calling `download-bundle` -needs `jpm/config/load-default` first (it sets `gitpath` and the cache dyns). - -## How it works - -`src/jolt/deps.janet` reads `deps.edn` (Janet parses it directly — EDN and Janet -syntax overlap for the `:deps`/`:paths` subset), then walks `:deps`: - -- `:git/url` (+ `:git/sha` or `:git/tag`) → `resolve-bundle` + `download-bundle` - into `jpm_tree/.cache`; +- `:git/url` + `:git/sha` (+ optional `:deps/root`) → clone the sha into the git + cache and contribute the checkout (or its `:deps/root` subdir); - `:local/root` → the path as-is; -- `:mvn/*` and anything else → ignored. +- `:mvn/*` → skipped with a warning; +- anything else → ignored. + +git resolution shells out to `git` through `jolt.host/sh` — `git init` + remote +add + fetch + reset at the requested sha. Clones land in a global, sha-immutable +cache (`$JOLT_GITLIBS`, else `~/.jolt/gitlibs`) shared across projects, the +`tools.gitlibs` `~/.gitlibs` model. Each resolved dependency contributes its own `:paths` (default `["src"]`) as source roots; the walk is **breadth-first** so every top-level coordinate registers before any transitive one — a top-level pin always wins, matching -tools.deps, and a coordinate conflict warns on stderr naming both. The result -is a de-duplicated, ordered list of directories. `resolve-deps-cached` memoizes -that list in the project-local `.cpcache/jolt-deps.jdn`, keyed on a hash of the -project `deps.edn` + the user-level `deps.edn` + the selected aliases. jpm is -loaded lazily (`require`, not `import`) so it's pulled in only when resolving — -never embedded in a built binary. +tools.deps. The result is a de-duplicated, ordered list of directories. -Three tools.deps features are mirrored in reduced form. **Aliases**: `:aliases` +Two tools.deps features are mirrored in reduced form. **Aliases**: `:aliases` entries supply `:extra-paths`/`:extra-deps` (accumulate across the aliases selected with `-A:a:b`) and `:main-opts` (last-wins, run with `-M:alias`). -**User config**: a `deps.edn` under `$JOLT_CONFIG` (else -`$XDG_CONFIG_HOME/jolt`, else `~/.jolt`) merges beneath the project file, -per key, project wins. **Tasks**: the honest subset of babashka's — a string -task is a shell command, a map task is `{:main-opts […] :doc "…"}`; bare -Clojure expressions aren't supported because the reader hands back parsed -data, and round-tripping it to source isn't worth the fragility. +**Tasks**: the honest subset of babashka's — a string task is a shell command, a +map task is `{:main-opts […]}`; bare Clojure expressions aren't a separate task +form. -Clones default to a global sha-immutable cache (`$JOLT_GITLIBS`, else -`/gitlibs`) shared across projects, the `tools.gitlibs` -`~/.gitlibs` model; per-project trees remain available by passing `tree` -explicitly. +## How the CLI ties it together -The loader (`evaluator.janet/find-ns-file`) resolves a namespace by searching the -context's `:source-paths` in order (the stdlib `src/jolt` first), trying `.clj` -then `.cljc`. Extra roots come from `JOLT_PATH` or `init`'s `:paths` option. +`jolt.main` (`jolt-core/jolt/main.clj`) is the CLI dispatch. Driven by `cli.ss`, +it resolves the project (`jolt.deps/resolve-project`), prepends the resolved +roots, and de-sugars the argv into a run: -The `jolt` CLI (`src/jolt/main.janet`, `resolve-deps-argv`) ties it together: on -a deps subcommand — or any runnable command in a directory that has a `deps.edn` -— it resolves the roots, sets `JOLT_PATH`/`JOLT_APP_PATHS`, and de-sugars the -argv into a plain runtime command (`-M:alias` → the alias `:main-opts`, `run -FILE` → `FILE`, …) that the normal dispatch then runs. `main.janet` imports -`deps.janet`, so the resolver ships in the `jolt` binary; but `deps.janet` loads -`jpm` lazily, and the runtime modules (`api`/`backend`/RT) never import it, so an -app baked from its own `jolt/api` entry doesn't link it. The runtime's only -dependency interface remains that one env var. +- `run -m NS args` → load `NS`, call its `-main`; +- `run FILE` → load the file; +- `-M:alias` → run the alias's `:main-opts`; +- `-A:alias` → add the alias's paths/deps, then run the rest; +- `repl` → a line REPL; +- `path` → print the resolved roots; +- `` → run a `deps.edn` `:tasks` entry. -`jolt uberscript` bundles a namespace and everything it requires into one -standalone `.clj`. It requires the entry namespace and uses the order in which -the loader finishes loading files — a dependency finishes before the file that -required it, so the order is topological — then concatenates that source. The -baked-in stdlib is excluded (it's part of the runtime, not bundled). - -Gotcha worth remembering: the `jolt` CLI's context is built into its image at -build time, so `JOLT_PATH` is applied at runtime in `main`, not in `init` (whose -env read would be frozen at build). +The resolver lives in the overlay alongside the runtime, but the runtime's only +dependency interface is the list of source roots it's handed. ## Limitations @@ -105,35 +71,9 @@ env read would be frozen at build). ## Conformance -`test/integration/deps-conformance-test.janet` resolves a few real pure-`cljc` -git libraries and reports whether their namespaces load and a sample call works. -It's network-gated behind `JOLT_CONFORMANCE=1` so CI stays offline. Use it to -check a library against the current interpreter, and to drive fixes for whatever -gap a failure points at (the same loop as the clojure-test-suite battery). A -library fails when it relies on something Jolt doesn't provide — JVM interop, or -a regex feature like Unicode property classes (`\p{…}`). - -## Not yet - -- **Compiling deps into a binary image.** `uberscript` already produces a - standalone `.clj`; baking a project's dependencies directly into a custom - executable image is a heavier variant that isn't implemented. - -## Janet dependencies: `:jpm/module` - -A jolt project can depend on janet libraries. jpm owns their installation; -`deps.edn` declares the requirement and `jolt` verifies it at resolve time: - -```clojure -:deps {janet/spork-http {:jpm/module "spork/http" - :jpm/install "spork"}} -``` - -- `:jpm/module` — the janet module path that must be importable. -- `:jpm/install` (optional) — the jpm package to install when it isn't; - `jolt` runs `jpm install ` once, then re-checks. Without it the resolve - fails with the install hint. - -A `:jpm/module` dep contributes no source roots. At runtime the `janet.*` -interop bridge autoloads the module on first reference -(`janet.spork.http/server`, …), so nothing else is needed. +The known-working libraries (see [libraries.md](libraries.md)) and the +[examples](https://github.com/jolt-lang/examples) exercise real pure-`cljc` git +libraries end to end — resolving them from git, loading their namespaces, and +running sample calls. A library fails when it relies on something Jolt doesn't +provide — JVM interop, or a regex feature like Unicode property classes +(`\p{…}`). diff --git a/foundational-runtime-handoff.md b/foundational-runtime-handoff.md deleted file mode 100644 index 31f123f..0000000 --- a/foundational-runtime-handoff.md +++ /dev/null @@ -1,198 +0,0 @@ -# Foundational Runtime Epic — Handoff - -**Epic:** jolt-5vsp · **Predecessor:** jolt-ffn (targeted specialization — concluded) -**Date:** 2026-06-16 - -This is a cold-start handoff. Read it top to bottom before touching code. Its -whole point is to keep the fresh session from re-running the experiments that -already came back flat, and to start from the one measurement that actually -tells us where to invest. - -## Why this epic exists - -The targeted-specialization epic (jolt-ffn) tried to close jolt's constant-factor -gap vs JVM Clojure with per-form compiler passes. Three independent attempts all -came back flat: - -| Attempt | Bead | Result | -|---|---|---| -| Record field-read guard removal (bare field reads) | jolt-3ko | ~3% on dispatch (shipped #141 — kept for correctness, not speed) | -| Protocol inline cache (runtime, per-method) | jolt-ez5h | ~0% — the per-dispatch gen-check exactly cancels the find-protocol-method saving; `find` was never the bottleneck | -| Record-ctor descriptor-baking (fewer allocs/record) | jolt-p7fo | flat on binary-trees + broke the gate; reverted | - -The conclusion: **the gap is structural to jolt-on-Janet, not a missing -optimization.** Targeted passes remove only the cheap parts; the structural floor -remains. - -## The scorecard (jolt / JVM Clojure) - -Regenerate any time with `JVM=1 bench/run.sh` (the absolute-reference mode). - -| Axis | Bench | jolt/JVM | -|---|---|---| -| Pure float compute | `mandelbrot` | **~15× ← THE FLOOR** | -| Persistent collections (HAMT) | `collections` | ~28× | -| Recursion (call + arith) | `fib` | ~37× | -| Megamorphic dispatch | `dispatch` | ~76× | -| Monomorphic dispatch | `mono-dispatch` | ~109× | -| Allocation / GC | `binary-trees` | ~314× (≈150× at depth 10) | - -`mandelbrot` is the floor: pure tight arithmetic loops — no dispatch, no -allocation, no collections — and native arith already fires (jolt-3pl). So ~15× -is what jolt's *execution substrate* costs on the simplest possible workload. -Every other axis adds structural overhead **on top** of that floor. - -**Machine caveat:** the dev machine swaps heavily (~13 GB). Alloc-heavy benches -(`binary-trees`, `collections`) inflate badly; light benches (`mandelbrot`, -`fib`, `dispatch`) are trustworthy. Get absolute alloc numbers on a clean machine. - -## The four structural walls - -1. **Bytecode-VM execution.** jolt's backend emits **Janet** (a register-bytecode - VM) and runs it on the Janet interpreter loop — no JIT, no native code. Every - op is bytecode dispatch. This is the `mandelbrot` 15× floor. -2. **Mark-sweep GC.** Janet's GC scans all live objects each cycle (no - generations). Live-data + alloc-heavy workloads (`binary-trees` retains the - tree) pay O(live) per GC. The JVM's generational GC makes young-object churn - nearly free. -3. **Indirect calls.** Protocol dispatch and fn calls go through indirection - (closures, the protocol registry). The JVM inlines/devirtualizes. jolt's - devirt (jolt-41m) only fires on *statically*-proven monomorphic sites; - `reduce`/`mapv` over a collection doesn't give that proof, so the common - runtime-monomorphic case pays full dispatch (that's why `mono-dispatch` is - *worse* than megamorphic — the JVM inline-caches it to near-free, jolt doesn't). -4. **Boxed / generic representations.** Records are tuples `[descriptor field…]`; - field access goes through a tag guard unless the type is proven. Generic ops - carry runtime type checks. (Open question: are Janet *numbers* boxed? Verify - in the spike — it decides whether unboxing is a lever or already done.) - -## Foundational levers (ranked) - -1. **Native codegen — emit C, not Janet bytecode.** The Stalin approach. Compile - jolt IR → C → machine code via the system compiler. The *only* lever that - moves the 15× compute floor; could approach C/JVM speed on compute-bound code. - Massive (a new backend). Plausible incremental shape: a jolt-IR→C compiler for - *hot* fns with a fallback to the existing bytecode path for unsupported forms — - mirroring today's interpret/compile hybrid. Needs to confirm Janet's C-API / - native-module story can be targeted incrementally. -2. **Structural GC-pressure reduction.** Value-type small records (avoid heap), - transient/editable-node hot paths (RFC 0003 future work — pvec/phm/sorted are - now tries/HAMT/RB, so O(1) `transient`/`persistent!` via editable nodes is - open). Helps the alloc-bound axes (`binary-trees`, `collections`). Does **not** - touch the compute floor. -3. **Deeper devirt + body inline.** Propagate element/return types so devirt - fires on runtime-monomorphic collections, then inline the method body - (jolt-4x9 element types + jolt-t6r). Helps dispatch. Bounded ceiling (still - bytecode underneath). - -## STATUS (2026-06-16) — lever 1 (native codegen) built and working - -The spike ran and lever 1 is now implemented. Full writeups: -`docs/foundational-runtime-spike-results.md` (floor localization) and -`docs/foundational-runtime-lever1-native-codegen.md` (native codegen). - -Done (all merged to main, PRs #143–#148): -- **Floor localized:** the 15.4× decomposes into a **Janet-VM floor ≈10.8× JVM** - (only native codegen moves it) + a **jolt loop-lowering ≈1.43×** (cheap backend - win, bead **jolt-v28u**). Janet numbers are already unboxed (not a lever). -- **Native codegen (jolt-ihdp, CLOSED):** `src/jolt/cgen.janet` translates - numeric-leaf fns (numeric in/out, native-op arithmetic + loop/recur/if/let/do) - to C. Wired into the backend `:def` emit under **`JOLT_CGEN=1`** (opt-in). The - `.so` is content-addressed + cached. **mandelbrot 224ms → 12.4ms (~18×)**, - beats JVM. Leaf-first falls out free (callers stay bytecode, call native fn). -- **Build-time AOT (jolt-a7ds, partial):** `:cgen-collect?` records leaf fns at - build, `aot-build` compiles them into one `.so` + manifest; `:cgen-prebuilt` + - `load-aot` install them at deploy with **no cc** (proven with cc off PATH). - -Open work under epic jolt-5vsp: -- **jolt-a7ds** — fuse the prebuilt `.so` + manifest into the `jpm` exe for a - literal single binary (+ a `jolt cgen-build -m app` CLI). The heaviest piece; - into jpm executable-build, not the compiler. -- **jolt-v28u** — `while`-loop lowering for tail `recur` (cheap ~30%, independent - of cgen; helps ALL loops, not just cgen candidates). -- **jolt-l1l4** — widen cgen numeric grammar (mod/rem/bit-ops/min/max, mixed fns). -- **jolt-qx70** — hot-fn auto-detection (drop the global `JOLT_CGEN` knob). -- Lever 2 (GC-pressure) and lever 3 (deeper devirt) — untouched; see below. - -The original spike instructions are preserved below for context. - -**Localize the 15× floor.** Build three `mandelbrot` implementations and compare: - -- **jolt-compiled** `mandelbrot` (already in `bench/mandelbrot.clj`), -- **hand-written Janet** `mandelbrot` (the same nested loop, idiomatic Janet — - write it directly, no jolt), -- **JVM Clojure** `mandelbrot`. - -Two ratios fall out: - -- **jolt-emitted-Janet vs hand-Janet** → how much overhead jolt's *backend* adds - over optimal Janet. To see jolt's emitted Janet, use the backend emit path - (`backend/emit-ir` on the analyzed `run`/`count-point` fns) — note `:arities` - etc. are jolt pvecs, so introspection is awkward; easier to read the emitted - Janet via the compile path or just A/B the timings. -- **hand-Janet vs JVM** → the Janet VM's own floor. - -Decision: - -- If **hand-Janet ≈ jolt** and hand-Janet is ~15× JVM → the floor is **Janet's - bytecode VM**. Native codegen (lever 1) is the only fix. Commit to the spike of - a jolt-IR→C path for one hot fn and measure. -- If **jolt ≫ hand-Janet** → jolt's backend emits suboptimal Janet; there's - headroom in the **backend** (cheaper, no new runtime). Find what it emits that - hand-Janet doesn't. - -Also measure the **GC share** on `binary-trees` (Janet GC stats around the run — -`(gccollect)` / `gcinterval`, or count allocations) to size lever 2 honestly. - -## Key files / mechanisms - -- **Backend (IR → Janet emit):** `src/jolt/backend.janet`. `native-ops` (~L322) - emits native Janet arith; `emit-ir` (~L674) runs passes then emits. A native-C - backend would branch here. -- **Passes / inference:** `jolt-core/jolt/passes.clj` (`run-passes`), - `jolt-core/jolt/passes/types.clj` (inference; the `:fn` branch ~L527 now seeds - ^Record param hints — #141), `jolt-core/jolt/passes/inline.clj` - (scalar-replace, `ctor-shape`). -- **Record representation:** `src/jolt/types_protocols.janet` — `make-record` - (~L145, the ~5-alloc/record path), `record-shape-for` (~L139, rebuilds its - cache key every call), `record-tag`. Records are tuples `[descriptor field…]`. -- **Dispatch + ctors:** `src/jolt/eval_runtime.janet` — - `protocol-dispatch-impl` (~L62), `make-deftype-ctor-impl` (~L382). -- **Config knobs:** `src/jolt/config.janet` — `JOLT_DIRECT_LINK`, - `JOLT_WHOLE_PROGRAM`, `JOLT_OPTIMIZE`, the `ctx-shaping-env-vars` list (any new - ctx-shaping env var MUST be added there and to `image-cache-path`). -- **Self-hosting design:** `docs/self-hosting-compiler.md` (the kernel/value-layer - boundary), `docs/rfc/0003-transients.md` (editable-node future work). - -## How to build, run, measure - -```sh -jpm build # build/jolt (ctx baked, ~20ms startup); from-source is ~8s cold -export PATH="$PWD/build:$PATH" -bench/run.sh # jolt only, WP on -JVM=1 bench/run.sh # jolt vs JVM scorecard (needs `clojure` on PATH) -bench/run.sh mandelbrot 400 # one bench, custom size -JOLT_WHOLE_PROGRAM=0 bench/run.sh # measure what WP buys -``` - -Gate: `jpm build; janet run-tests.janet` (parallel, ~100s; `JOLT_TEST_JOBS` -overrides). Bench memory hygiene (`bd memories bench-isolation-gotcha`): never run -a perf matrix while other CPU work runs — it starves later configs and produces -bogus numbers. Sandwich A/B/A. - -## What NOT to repeat (already flat — see beads for detail) - -- Runtime protocol inline cache (jolt-ez5h): gen-check cancels the saving. -- Field-read guard removal as a *speed* play (jolt-3ko): ~3%; machinery dominates. - (The #141 change is kept for correctness + the `with-meta`-on-symbols fix.) -- `make-record` descriptor-baking (jolt-p7fo): flat — `binary-trees` is dominated - by the live retained tree + GC, not the short-lived intermediate allocs. - -## Open questions for the spike - -- Are Janet numbers boxed? (Lever or already done.) -- Does Janet expose a native-module / C-codegen path jolt can target incrementally - (hot fns → C, rest → bytecode)? -- What fraction of `binary-trees` is GC vs execution? -- Is there a cheaper record representation (Janet struct vs tuple-with-descriptor) - that lowers field-read + alloc cost without a new backend? diff --git a/host/chez/http-server.ss b/host/chez/http-server.ss index b59995d..c4e129d 100644 --- a/host/chez/http-server.ss +++ b/host/chez/http-server.ss @@ -3,8 +3,8 @@ ;; connection handled at a time on a background accept thread; synchronous Ring ;; handlers. Enough to serve a small web app. ;; -;; Exposed as jolt.http.server/run-server and (for the ring-app example) as -;; ring-janet.adapter/run-server + stop-server — both baked namespaces. +;; Exposed as jolt.http.server/run-server + stop-server — a baked namespace an +;; app requires for a Ring-style adapter (run a handler, stop the server). (load-shared-object #f) ; resolve socket/bind/listen/accept/recv/send in the process @@ -227,6 +227,3 @@ (def-var! "jolt.http.server" "run-server" (lambda (handler . opt) (http-run-server handler (if (pair? opt) (car opt) (jolt-hash-map))))) (def-var! "jolt.http.server" "stop-server" http-stop-server) -;; the ring-app example reaches the server through this adapter namespace. -(def-var! "ring-janet.adapter" "run-server" (lambda (handler . opt) (http-run-server handler (if (pair? opt) (car opt) (jolt-hash-map))))) -(def-var! "ring-janet.adapter" "stop-server" http-stop-server) diff --git a/host/chez/seed/README.md b/host/chez/seed/README.md index e4eccb7..65833b8 100644 --- a/host/chez/seed/README.md +++ b/host/chez/seed/README.md @@ -1,7 +1,7 @@ # Chez bootstrap seed -These two files are the **bootstrap compiler** for jolt-on-Chez — the seed that -makes the build self-hosting with no Janet in the loop: +These two files are the **bootstrap compiler** for jolt — the seed that makes +the build self-hosting: - `prelude.ss` — the `clojure.core` prelude (all tiers + clojure.string/walk/ template/edn/set/pprint) as Scheme `def-var!` forms. diff --git a/spike/chez/RESULTS.md b/spike/chez/RESULTS.md deleted file mode 100644 index 4ab415d..0000000 --- a/spike/chez/RESULTS.md +++ /dev/null @@ -1,124 +0,0 @@ -# Chez Scheme re-host spike — results - -Branch `spike/chez-bootstrap`. Question: would re-hosting jolt's substrate from -Janet onto Chez Scheme (cisco/ChezScheme 10.4.1) buy speed, at what size/memory -cost? This spike does NOT port jolt-core/RT — it measures the **execution -substrate ceiling** by hand-translating the two compute-bound benches (fib, -mandelbrot) into the Scheme a jolt->Chez backend would emit, plus real -size/memory of the Chez runtime. - -Machine: darwin arm64, M-series. Same caveat as the handoff doc — this dev box -swaps under load, so alloc-heavy absolute numbers inflate; compute benches -(fib/mandelbrot) are trustworthy. All runs isolated (no other CPU work). - -## Speed (mean ms, 3 runs after warmup; same sizes as bench/run.sh) - -| Bench | Janet jolt | Chez best | Speedup | Note | -|-------------------------------|-----------:|----------:|--------:|------| -| fib 30 | 246.6 | 5.2 | ~47x | fixnum arith — immediate, unboxed | -| mandelbrot 200 (generic ops) | 166.3 | 98.1 | ~1.7x | `+ - * >` box every flonum | -| mandelbrot 200 (flonum ops) | 166.3 | 13.4 | ~12.4x | `fl*/fl+/fl<` unboxed | - -Correctness verified: fib 30 = 832040, mandelbrot 200 count = 3288753 (both -match jolt). optimize-level 2 vs 3 made no material difference here. - -**The key finding** is the mandelbrot split. Generic Scheme arithmetic on floats -sends Chez through the numeric tower and **heap-boxes every flonum** — so the -naive emit gets almost nothing (~1.7x) and opt-level doesn't help. Emitting -flonum-specific ops (`fl+`/`fl*`/`fl<`, `fx` for the integer counter) lets Chez -keep flonums unboxed in registers and the same code drops to 13.4 ms. - -13.4 ms ~= jolt's own JOLT_CGEN C-codegen result (12.4 ms, which already beat -JVM per docs/foundational-runtime-lever1-native-codegen.md). So **Chez's native -compiler reaches the hand-emitted-C ceiling on its own**, with no separate `cc` -step, no `.so` cache, no AOT manifest — just runtime compilation, REPL intact. - -Implication for a real backend: the win is gated on the same type-inference -> -specialized-op lowering jolt ALREADY has (passes/types.clj feeds native-arith on -Janet today). fib's 47x is free (fixnums); mandelbrot's 12x needs that typed -path wired to `fl*` emission instead of (or alongside) the Janet/C path. - -## Size (deployable footprint) - -App code is negligible — fib compiled to a native object (`compile-program`, -optimize-level 3) is **2 KB**. The footprint is the Chez runtime: - -| Artifact | Size | vs Janet | -|---------------------------------------------------|--------:|---------:| -| Janet `build/jolt` (complete, jolt baked in) | 2.21 MB | 1.0x | -| Chez base, AOT (kernel + petite.boot + app) | 2.89 MB | 1.3x | -| Chez base, dynamic/REPL (+ scheme.boot compiler) | 3.96 MB | 1.8x | - -components: libkernel.a 0.83 MB, petite.boot (runtime lib) 2.07 MB, scheme.boot -(compiler) 1.07 MB. - -Caveat: the Chez rows are the runtime base ONLY. A complete jolt adds compiled -jolt-core (analyzer + clojure.core + persistent-collection RT) on top, which the -Janet 2.21 MB already includes. Estimated full Chez jolt ~4-6 MB. Still -single-digit MB, ~2-3x Janet, vastly under a JVM (40 MB+). petite.boot carries -much jolt won't use; a stripped custom boot file could shrink it. - -## Memory (max RSS) - -| Scenario | Janet | Chez | -|-----------------------------------|--------:|----------------:| -| startup / trivial | 12.5 MB | 32.1 (petite) / 49.5 (full) | -| mandelbrot 200 | 20.8 MB | ~32 MB (AOT under petite) | -| fib 30 | 19.8 MB | 32.1 MB | - -Chez's baseline is flat across workloads (fib allocates ~nothing and doesn't -move it), so the ~32 MB (runtime) / ~49.5 MB (runtime + resident compiler) is -**fixed reservation**, not workload allocation. This is the one axis where Chez -is clearly worse: ~2.5x Janet's fixed footprint. Trades RAM for speed. -(Potentially tunable via Chez heap params / a stripped boot file; not explored.) - -## Verdict - -- **Speed: validated and large on compute** — 47x (fib) and 12.4x (mandelbrot), - the latter matching jolt's C-codegen ceiling, **conditional** on the backend - emitting typed/specialized numeric ops. Naive generic emit is nearly flat on - floats. jolt's existing type passes are the lever that makes this real. -- **Chez could subsume the cgen path:** runtime native compile gets C-level - numeric speed while keeping live redefinition — collapsing the - interpret/compile/cgen-to-C hybrid into one native path. -- **Size: fine** (~1.3-1.8x base, ~2-3x full; single-digit MB). -- **Memory: the cost** (~2.5x fixed baseline). - -## Phase 1 — real-pipeline measurement (2026-06-18) - -The numbers above are hand-translated Scheme (the substrate ceiling). Phase 1 -(jolt-cf1q.2) ran the SAME benches end to end through the real pipeline (Clojure -source -> Janet-hosted analyzer -> IR -> Scheme emitter -> Chez compile), timed -in-process (`test/chez/bench-pipeline.janet`, Chez startup excluded): - -| bench | real pipeline | ceiling (this run) | gap = Phase 4 lever | -|---------------------|---------------|---------------------------|---------------------| -| fib 30 (flonum) | 14.4 ms | 7.1 ms hand-flonum | 2.0x dispatch/var | -| fib 30 (vs fixnum) | 14.4 ms | 5.2 ms fixnum | all-double model | -| mandelbrot 200 | 87.3 ms | 98.1 ms generic-flonum | AT/below ceiling | -| mandelbrot 200 typed| 87.3 ms | 13.4 ms typed fl*/fx* | typed emit (Phase 4)| - -Findings: (1) **compile-only is total** for the compute subset — every form -emits, no interpreter fallback (Chez has none). (2) Mandelbrot through the real -pipeline runs AT the generic-flonum ceiling (87 vs 98 ms) — the substrate ceiling -is reached end-to-end. (3) The fib residual is jolt's all-double number model -(the spike's 5.2 ms fib is fixnum); closing it to the 13.4 ms / fixnum ceiling is -the typed fl*/fx* emission Phase 4 owns. Eliding the redundant `jolt-truthy?` -wrapper on boolean-test `if`s (jolt-nkcb) cut fib 24.0 -> 14.4 ms. - -## NOT yet measured (needs the RT port — the real project, not a spike) - -- collections / binary-trees: these hit persistent collections + GC. Chez's GC - is **generational** (vs Janet's non-generational mark-sweep), so binary-trees - (jolt's worst axis, ~314x JVM) is exactly where Chez's GC should help most — - but it requires porting the persistent-collection RT first. This is the next - validation and the highest-uncertainty remaining question. -- Startup time (Janet jolt baked-image ~20ms; Chez boot-file load TBD). -- fiber/async layer (Janet fibers -> call/cc + threads rebuild). - -## Repro - - cd spike/chez - chez --script fib.ss 30 3 - chez --script mandelbrot.ss 200 3 # generic (boxed) — slow - chez --script mandelbrot-fl.ss 200 3 # flonum-typed — the ceiling diff --git a/test/chez/README.md b/test/chez/README.md index db51aad..46a8a23 100644 --- a/test/chez/README.md +++ b/test/chez/README.md @@ -1,6 +1,6 @@ # Chez test harness -The correctness gate for jolt. Pure Chez (+ Clojure for the JVM oracle), no Janet. +The correctness gate for jolt. Pure Chez (+ Clojure for the JVM oracle). Correctness is judged against the JVM-sourced conformance spec; the spec itself lives in `test/conformance/` (see its `SPEC.md`). Run the whole gate with `make test` from the repo root. diff --git a/test/conformance/README.md b/test/conformance/README.md index 6d4e6ac..2e5ffa6 100644 --- a/test/conformance/README.md +++ b/test/conformance/README.md @@ -7,13 +7,14 @@ The corpus (`test/chez/corpus.edn`) is jolt's host-neutral behavioral suite — one row per case: `{:suite :label :expected :actual}`, where `:actual` is a Clojure -source expression and `:expected` its result (or `:throws`). Runtime harnesses -(`test/chez/run-corpus-prelude.janet`, `run-corpus-zero-janet.janet`) replay it on -each host and compare by value-equality. +source expression and `:expected` its result (or `:throws`). The runtime harness +(`host/chez/run-corpus.ss`, invoked by `make corpus`) replays it on Chez and +compares by value-equality. -Historically every `:expected` was **hand-written by jolt developers**. That makes -the corpus a fine regression suite but a weak *specification*: it certifies jolt -against its authors' beliefs, not against Clojure. This directory closes that gap. +Every `:expected` is sourced from reference JVM Clojure, so the corpus is both a +regression suite and a *specification* certified against Clojure rather than +against its authors' beliefs. This directory holds the certification tooling that +closes that gap. ## What's here @@ -34,17 +35,17 @@ against its authors' beliefs, not against Clojure. This directory closes that ga entries with a tracked bead. These categories become the `:features` flags in conformance inc3. -- **`certify-test.janet`** — gate wrapper. Skips cleanly when `clojure` (JVM) is - not installed; otherwise runs `certify.clj` and fails the build on a **NEW** - (unclassified) divergence or a **stale** allowlist entry. Flaky entries (JVM - result is timing-dependent, e.g. `future-cancel`) are tolerated either way. +`make certify` is the gate wrapper. It skips cleanly when `clojure` (JVM) is not +installed; otherwise it runs `certify.clj` and fails the build on a **NEW** +(unclassified) divergence or a **stale** allowlist entry. Flaky entries (JVM +result is timing-dependent, e.g. `future-cancel`) are tolerated either way. ## Running ```sh -clojure -M test/conformance/certify.clj # gate (exit≠0 on new/stale) +make certify # the gate wrapper (skips if clojure absent) +clojure -M test/conformance/certify.clj # gate directly (exit≠0 on new/stale) clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn /tmp/report.edn # full machine-readable report -janet test/conformance/certify-test.janet # the gate wrapper ``` ## Current state diff --git a/test/conformance/SPEC.md b/test/conformance/SPEC.md index 33ca5a4..b4a575d 100644 --- a/test/conformance/SPEC.md +++ b/test/conformance/SPEC.md @@ -70,7 +70,6 @@ implements. Current profile (≈2670 portable, ≈249 non-portable): | `:concurrency/snapshot` | isolated-heap futures/agents/pmap — captured atoms are snapshotted, not shared | | `:host/jvm-interop` | Java classes / `instance?` on host classes / proxy / bean / definterface | | `:host/arrays` | Java arrays (`into-array`, `int-array`, …) | -| `:host/janet` | Janet host interop (`janet.*`) | | `:async/core-async` | `clojure.core.async` channels/`go` | | `:runtime/eval` | runtime `eval` / `load-string` | | `:reader/jolt` | jolt reader features (`#?(:jolt …)`) + syntax-quote literal collapse |