simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: consolidate doc/ into docs/

Browse source 
One documentation root: the prose docs (building-and-deps, self-hosting
architecture/compiler, tools-deps, grammar.ebnf) join the spec and RFCs under
docs/. References in README and deps-conformance-test updated.
This commit is contained in:
Yogthos 2026-06-10 12:10:28 -04:00
parent fdfd086df6
commit 808ce6a725
5 changed files with 0 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

117
docs/building-and-deps.md Normal file
View file

@ -0,0 +1,117 @@
# Building and dependencies
How to build Jolt from source and how to pull Clojure libraries into a project.
## Building
```bash
git clone https://github.com/jolt-lang/jolt.git
cd jolt
git submodule update --init # vendor/sci (used by the SCI bootstrap tests)
jpm build
```
This produces two executables under `build/`:
- **`jolt`** — the runtime: REPL, file/expr runner, nREPL server. The whole `.clj`
standard library (`clojure.string`/`set`/`walk`/`edn`/`zip`, `jolt.http`/
`interop`/`shell`/`nrepl`) is baked into this binary at build time, so it loads
from any directory — the build artifact is self-contained. (`clojure.core` is
built into the runtime in Janet and auto-referred, so it's always available.)
- **`jolt-deps`** — a separate tool that resolves a `deps.edn` (see below). It
sits beside the runtime the way `jpm` sits beside `janet`; the runtime itself
knows nothing about deps.edn.
Needs `jpm` and a recent Janet — developed and CI-tested against **1.41**. The
futures and core.async layers use Janet's threaded `ev/` channels (`ev/thread`,
`ev/thread-chan`), so older Janets may not run the full suite.
`jpm build` doesn't always notice source changes; run `jpm clean && jpm build`
after editing `src/` to be sure the binaries are current. `jpm test` runs against
the source directly, so it never goes stale.
## How namespaces are found
`(require ...)` resolves a namespace to a file by searching an ordered list of
source roots — the stdlib first, then any extra roots — trying `<ns>.clj` then
`<ns>.cljc` (dots become directories, dashes become underscores). Extra roots
come from:
- `JOLT_PATH` — a colon-separated list of directories (like a classpath), applied
at runtime;
- the `:paths` option to `init` when embedding Jolt as a library.
If a namespace isn't found on any root, the loader falls back to the stdlib baked
into the binary — that's how `clojure.string` and friends resolve when you run
the binary outside the source tree.
So you can point Jolt at a directory of Clojure source with no deps machinery at
all:
```bash
JOLT_PATH=/path/to/lib/src build/jolt myfile.clj
```
## Dependencies via deps.edn
`jolt-deps` reads a `deps.edn` in the current directory, fetches its
dependencies, and runs `jolt` with the resolved source directories on
`JOLT_PATH`.
```bash
jolt-deps path # print the resolved roots (':'-joined)
jolt-deps run FILE [args] # resolve, then run `jolt FILE …`
jolt-deps repl # resolve, then start a REPL
jolt-deps -e EXPR [args] # resolve, then evaluate EXPR
```
`jolt-deps` launches the `jolt` binary it finds on `PATH` (override with
`$JOLT_BIN`).
Example `deps.edn`:
```clojure
{:paths ["src"]
:deps {weavejester/medley {:git/url "https://github.com/weavejester/medley"
:git/tag "1.0.0"}
my/helpers {:local/root "../helpers"}}}
```
```bash
jolt-deps run -m myapp.main
```
### What's supported
- **git deps** — `{:git/url … :git/tag …}` or `{:git/url … :git/sha …}` (use a
full SHA; `git fetch` can't resolve a short one). Transitive deps from each
dependency's own `deps.edn` are resolved too.
- **local deps** — `{:local/root "../path"}`.
- The project's own `:paths` (default `["src"]`) are included.
Resolution reuses jpm's git fetch and cache (a dependency is cloned once into
`jpm_tree/.cache` and reused). Resolved roots are cached on a hash of `deps.edn`,
so an unchanged `deps.edn` doesn't re-fetch.
### What's not
- **No Maven.** `:mvn/version` deps are ignored — git and local only.
- **Pure `clj`/`cljc` only.** A library that needs the JVM (Java interop, host
classes) or a `clojure.core` feature Jolt doesn't implement will fail to load
or fail at a call. Coverage is per-function: a namespace can load with most
functions working and a few not.
### Bundling into one file
`jolt uberscript OUT.clj -m NS` (or `jolt-deps uberscript …`, which resolves deps
first) bundles `NS` and every namespace it requires — your code plus its
dependencies — into a single `.clj` in dependency order, ending with a call to
`NS/-main`. The result runs on a plain `jolt` with no `JOLT_PATH`, no deps
fetched, and no jpm:
```bash
jolt-deps uberscript app.clj -m myapp.main
jolt app.clj arg1 arg2
```
See [`tools-deps.md`](tools-deps.md) for the design rationale.

185
docs/grammar.ebnf Normal file
View file

@ -0,0 +1,185 @@
(* ===========================================================================
Jolt reader grammar (EBNF)
===========================================================================
This grammar specifies the surface syntax accepted by Jolt's reader
(src/jolt/reader.janet) — the text that `read`/`parse-string`/`load-string`
turn into data/forms. It is the syntactic half of Jolt's contract; the
behavioural half lives in test/spec/. Where Jolt diverges from Clojure the
difference is called out in a comment.
Notation (ISO-ish EBNF):
= definition | alternation
, concatenation ( ) grouping
[ x ] optional (0 or 1) { x } repetition (0 or more)
"x" 'x' terminal (* *) comment
? x ? informal/char-class terminal
A source file is a sequence of forms; evaluation is form-by-form.
=========================================================================== *)
file = { ws | form } ;
(* --------------------------------------------------------------------------
Whitespace, separators and comments
-------------------------------------------------------------------------- *)
ws = { whitespace-char | comment | discard } ;
whitespace-char = " " | "\t" | "\n" | "\r" | "," ; (* comma is whitespace *)
comment = ";" , { ? any char except "\n" ? } , [ "\n" ] ;
discard = "#_" , ws , form ; (* read then drop next form *)
(* --------------------------------------------------------------------------
Forms
-------------------------------------------------------------------------- *)
form = scalar
| collection
| reader-macro
| dispatch ;
scalar = nil | boolean | number | string | character
| keyword | symbol ;
collection = list | vector | map ;
(* --------------------------------------------------------------------------
Atoms / scalars
-------------------------------------------------------------------------- *)
nil = "nil" ;
boolean = "true" | "false" ;
(* Numbers. Jolt accepts Clojure's numeric literal syntaxes, but — since Jolt
numbers are Janet ints/doubles — it has no distinct bignum, ratio or
BigDecimal types: the BigInt suffix N and BigDecimal suffix M are read as the
plain number, a ratio a/b is read as its double quotient, and radixed
integers are computed by base. The symbolic floats ##Inf/##-Inf/##NaN are
also read. (No octal-with-leading-0 literal.) *)
number = symbolic-value
| [ sign ] , ( radix-int | ratio | hex-int | decimal ) ;
sign = "+" | "-" ;
integer = digit , { digit } ;
hex-int = "0" , ( "x" | "X" ) , hex-digit , { hex-digit } , [ "N" ] ;
radix-int = integer , ( "r" | "R" ) , alnum , { alnum } ; (* base 2..36: 2r1010, 16rFF, 36rZ *)
ratio = integer , "/" , integer ; (* read as a double quotient *)
decimal = integer , [ "." , digit , { digit } ] , [ exponent ] , [ num-suffix ] ;
exponent = ( "e" | "E" ) , [ sign ] , digit , { digit } ;
num-suffix = "N" | "M" ; (* BigInt / BigDecimal in Clojure; plain number in Jolt *)
symbolic-value = "##Inf" | "##-Inf" | "##NaN" ;
digit = "0".."9" ;
hex-digit = digit | "a".."f" | "A".."F" ;
alnum = digit | "a".."z" | "A".."Z" ;
(* Strings: double-quoted with backslash escapes. *)
string = '"' , { string-char } , '"' ;
string-char = ? any char except '"' and "\" ? | escape ;
escape = "\" , ( "n" | "t" | "r" | "\" | '"' | ? other ? ) ;
(* Characters: \x, a named char, or \uNNNN / \oNNN. *)
character = "\" , ( char-name | ? any single char ? ) ;
char-name = "newline" | "space" | "tab" | "return"
| "formfeed" | "backspace" | "nul"
| "u" , hex-digit , hex-digit , hex-digit , hex-digit
| "o" , octal-digit , { octal-digit } ;
octal-digit = "0".."7" ;
(* Keywords: :name, :ns/name, or auto-resolved ::name. *)
keyword = "::" , sym-token (* auto-resolved to current ns *)
| ":" , sym-token ;
(* Symbols: a token of symbol-constituent chars not parsed as a number. A "/"
separates an optional namespace from the name. *)
symbol = sym-token ;
sym-token = sym-start , { sym-constituent } ;
sym-start = sym-constituent - ( digit ) ; (* not a leading digit *)
sym-constituent = letter | digit
| "*" | "+" | "!" | "_" | "-" | "?" | "." | "<" | ">"
| "=" | "&" | "|" | "$" | "%" | "/" ;
letter = "a".."z" | "A".."Z" ;
(* --------------------------------------------------------------------------
Collections
-------------------------------------------------------------------------- *)
list = "(" , { ws | form } , ")" ; (* -> a Clojure list *)
vector = "[" , { ws | form } , "]" ; (* -> a persistent vector *)
map = "{" , { ws | map-entry } , "}" ;
map-entry = form , ws , form ; (* an even number of forms *)
(* --------------------------------------------------------------------------
Reader macros (prefix sugar). Each expands to a 2-element form
(op operand), e.g. 'x -> (quote x), @a -> (deref a).
-------------------------------------------------------------------------- *)
reader-macro = quote | syntax-quote | unquote | unquote-splice
| deref | metadata ;
quote = "'" , form ; (* (quote form) *)
syntax-quote = "`" , form ; (* (syntax-quote form) *)
unquote-splice = "~@" , form ; (* (unquote-splicing form) *)
unquote = "~" , form ; (* (unquote form) *)
deref = "@" , form ; (* (deref form) *)
metadata = "^" , meta-form , ws , form ; (* attach metadata to form *)
meta-form = map | keyword | symbol | string ;
(* Normalized like Clojure: a symbol or string is a type hint -> {:tag ...};
a keyword -> {keyword true}; a map is used as-is. On a symbol the metadata
rides on the symbol (it stays a bare symbol, so a hint like ^String is
transparent in params/lets/bodies); other targets use a runtime with-meta. *)
(* --------------------------------------------------------------------------
Dispatch forms — introduced by "#".
-------------------------------------------------------------------------- *)
dispatch = set
| anon-fn
| var-quote
| regex
| reader-conditional
| discard (* #_form — see Whitespace above *)
| tagged-literal ;
set = "#{" , { ws | form } , "}" ; (* a persistent set *)
(* Anonymous function. %, %1.. are positional params; %& is the rest param. *)
anon-fn = "#(" , { ws | form } , ")" ; (* -> (fn* [..] body) *)
anon-arg = "%" | "%" , digit , { digit } | "%&" ;
var-quote = "#'" , symbol ; (* (var symbol) *)
(* Regex literal -> a Janet PEG-backed regex value.
Supported: groups, greedy/lazy quantifiers, (?:..), lookahead (?=..)/(?!..),
alternation, anchors ^ $ \b \B, classes, (?i). NOT: lookbehind,
backreferences, named groups. *)
regex = '#"' , { ? any char except '"' ? | "\" , ? any char ? } , '"' ;
(* Reader conditional. Jolt reads as the :clj platform; #?@ splices a sequence. *)
reader-conditional
= "#?@" , "(" , { ws | feature , ws , form } , ")"
| "#?" , "(" , { ws | feature , ws , form } , ")" ;
feature = ":clj" | ":cljs" | ":default" | keyword ;
(* Tagged literal: #tag form. Built-in tags include #inst and #uuid; others
dispatch to a registered data-reader (else error). *)
tagged-literal = "#" , symbol , ws , form ;
(* ── Destructuring (semantics, not reader syntax) ────────────────────────────
The reader produces plain vectors and maps; the binding MACROS (let, fn, loop,
doseq, for, defmacro params, …) interpret them as destructuring patterns. The
PRIMITIVES they desugar to — fn*, let*, loop* — take plain symbols ONLY (a
non-symbol binding is an error, as in Clojure: "fn params must be Symbols" /
"Bad binding form, expected symbol"). The grammar of a pattern:
binding = symbol | seq-binding | map-binding ;
seq-binding = "[" , { binding } , [ "&" , binding ] , [ ":as" , symbol ] , "]" ;
map-binding = "{" , { binding , form (* {local key} *)
| ":keys" , "[" {symbol} "]" (* x or ns/x *)
| ":strs" , "[" {symbol} "]"
| ":syms" , "[" {symbol} "]"
| ":or" , map (* defaults *)
| ":as" , symbol } , "}" ;
A symbol in :keys/:syms may be namespaced (x/y): the namespaced key is looked
up but the local is the bare name (y). Destructuring a *sequence* with a
map-binding treats it as keyword args (alternating k v, or a trailing map) —
this is the fn/macro `[& {:keys [...]}]` form. *)

138
docs/self-hosting-architecture.md Normal file
View file

@ -0,0 +1,138 @@
# Self-hosting architecture: portable jolt-core over a host runtime
Design for splitting Jolt into a **portable Clojure-in-Clojure core** and a
**host runtime** (Janet today, another runtime tomorrow), so the language is
truly self-hosted and `jolt-core` can be lifted out and re-hosted.
This is the design that must be right *before* writing the compiler in Clojure —
see [[self-hosting-compiler]] for the staged plan it plugs into.
## What "truly self-hosted + portable" requires
Two independent properties:
1. **Self-hosted** — the compiler and most of `clojure.core` are written in
Clojure and compiled by Jolt itself.
2. **Portable** — that Clojure code (`jolt-core`) depends only on a small,
explicit **host contract**, never on Janet directly. Re-hosting means
implementing the contract for a new runtime; `jolt-core` is reused verbatim.
The enemy is `jolt-core` calling `janet/tuple`, `make-vec`, `ns-find`, etc.
directly — that welds it to Janet. Every host dependency must go through the
contract.
## Prior art (the seam everyone uses)
- **Clojure (JVM).** `clojure.lang.*` (Java) is the host: `RT`/`Numbers` runtime
helpers, the `Compiler` (form → JVM bytecode), persistent data structures,
`Var`/`Namespace`, the reader. `clojure/core.clj` is the language, in Clojure.
Seam: ~20 primitive special forms + `RT` static methods. Everything else is
Clojure.
- **ClojureScript (self-hosted).** Two portable passes — `cljs.analyzer`
(form → AST **as data**, reading a **compiler-state map** of
namespaces/defs/macros, *not* host objects) and `cljs.compiler` (AST → JS, the
host-specific back end). `cljs.core` is Clojure compiled to JS. Platform splits
live in `.cljc` reader conditionals. This is the closest model to what we want:
**the analyzer is host-agnostic; only the back end and the runtime are
host-specific.**
- **Nanopass / Guile Tree-IL.** A high-level IR is the portability seam; multiple
back ends consume it.
- **ClojureCLR / ClojureDart / jank.** Same shape every time: portable analyzer +
host back end + host runtime.
The invariant across all of them: **the IR (analyzer output) and a small runtime
protocol are the contract; the front end is portable, the back end and runtime
are per-host.**
## Decisions (locked)
- **Seam = a minimal host protocol.** `jolt-core` calls a small documented set of
host fns (in ns `jolt.host`): `resolve-sym`, `macro?`, `macroexpand-1`,
`current-ns`, `intern!`, plus the `RT` primitives. Each host provides `jolt.host`
(+ RT). Re-hosting = reimplement that handful of fns. The protocol *is* the
boundary; `jolt-core` never touches Janet directly.
- **Physical split now.** Portable Clojure lives under `jolt-core/` (a new source
root, embedded into the binary like the rest of the stdlib); host Janet code for
the new pipeline under `host/janet/`. Legacy host modules under `src/jolt/*.janet`
are the existing Janet host and get relocated under `host/janet/` in a later
mechanical pass (tracked) — not moved big-bang now, to keep the suite green.
## The Jolt split
```
jolt-core/ PORTABLE Clojure — no Janet. Depends only on the contract.
ir the IR spec (data shapes the analyzer emits)
analyzer form -> IR (macroexpands; resolves via host protocol)
macros when/cond/->/defn/... (the macro library, in Clojure)
core clojure.core fns expressible in Clojure, over RT primitives
host/janet/ THE HOST — Janet. Implements the contract.
reader text -> jolt forms
rt data structures + RT primitive fns (cons/first/+/get/apply…)
backend IR -> Janet forms -> Janet compile -> bytecode (the emitter)
cenv the compile-time host protocol impl (resolve/macro?/intern)
bootstrap load jolt-core, wire analyzer+backend into the loader
interop janet.* bridge
```
Two contracts cross the seam:
### 1. The IR (analyzer → back end)
The existing `:op`-tagged AST, made **host-neutral**:
- `{:op :const :val v}`, `:if`, `:do`, `:let`, `:fn` (arities), `:invoke`,
`:vector`/`:map`/`:set`, `:quote`, `:throw`/`:try`, `:loop`/`:recur`.
- **Globals reference vars by NAME, not by host cell:**
`{:op :var :ns "clojure.core" :name "map"}`. (compiler.janet today embeds the
Janet var cell as a constant — that's a host leak and breaks AOT. Name-based
refs are both portable and AOT-friendly; the back end resolves the cell.)
- No embedded host function values. Calls to runtime primitives are
`{:op :rt :name "cons"}` resolved by the back end to the host's RT fn.
### 2. The host contract (two protocols)
- **Compile-time (`cenv`)** — what the analyzer needs from the host while
analyzing: `(current-ns)`, `(resolve-sym sym) -> {:kind :var|:macro|:local|:special|:host, :ns, :name}`,
`(macroexpand-1 form)`, `(intern! ns sym meta)`. The analyzer calls only these;
it never touches Janet ns/var tables. (CLJS keeps this as pure data; we use a
small protocol — a minimal, documented boundary — because Jolt already has live
ns/var objects. The protocol *is* the seam.)
- **Runtime (`RT`)** — the primitive fns emitted code and `jolt-core` call by
stable name: arithmetic/compare, `cons/first/rest/seq/conj/get/assoc/count`,
`apply`, `=`, vector/map/set constructors, var deref/bind, keyword/symbol
construction. The back end maps each to the host (on Janet, mostly the existing
`core-*`). To re-host, implement this set.
## Why name-based vars (not embedded cells)
`compiler.janet` compiles a global ref to a closure over the Janet var cell. That
(a) is a Janet value baked into the IR — not portable, and (b) can't be marshaled
for AOT without the runtime-dict trick. Compiling instead to *resolve var by
(ns,name) at call time* through an RT primitive keeps redefinition live, makes the
IR host-neutral, and makes images trivially portable. The per-call lookup is the
cost; it can be cached/direct-linked later as an opt-in optimization.
## Bootstrap & staging (keeps the suite green throughout)
`compiler.janet` stays as the **bootstrap back end** until the Clojure pipeline is
proven. Order:
1. **Freeze the IR** spec and refactor `compiler.janet`'s emit to consume
name-based `:var` (no behavior change; bootstrap still works).
2. **Define the host contract** (`cenv` + `RT`) and implement it on Janet,
exposed under a stable namespace the Clojure core can call.
3. **Write `jolt.analyzer` in Clojure** producing IR, against `cenv`. Diff its IR
against the Janet analyzer on the conformance corpus until identical.
4. **Janet back end consumes IR** from the Clojure analyzer; wire into the loader
behind a flag. Validate at parity (dual-mode conformance + clojure-test-suite).
5. **Flip** the loader to the Clojure analyzer + Janet back end; `compiler.janet`
shrinks to the back end only.
6. **Move `clojure.core`** macros then fns into `jolt-core` incrementally, each
compiled by the prior stage, isolating host bits behind `RT`.
Guards at every step: the dual-mode conformance harness (interpret vs compile)
and the clojure-test-suite baseline.
## The portability test
When done, re-hosting Jolt to runtime X means writing only: `host/X/{reader, rt,
backend, cenv, bootstrap}`. `jolt-core/{ir, analyzer, macros, core}` is reused
unchanged. That is the concrete bar for "truly self-hosted and portable."

172
docs/self-hosting-compiler.md Normal file
View file

@ -0,0 +1,172 @@
# Toward a self-hosting Jolt compiler
Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc
compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet
bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet
bootstrap. This is a design doc, not a changelog — it describes where we are, the
prior art, the constraints we verified, and a recommended path.
## The goal
- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is
enough to start; the rest of Clojure — including the compiler — is written in
Clojure and compiled by Jolt itself, growing the language as it compiles more
of itself.
- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast),
not tree-walking.
- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols,
multimethods, and everything else stay live and redefinable at the REPL even
for compiled code.
- **Minimal host requirement.** Shrink what must exist in Janet to the
irreducible base.
## Where Jolt is today
- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a
tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the
stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in
the host, inverted from the Clojure-in-Clojure ideal.
- The interpreter (`eval-form`) is the complete reference path.
- The compiler (`compiler.janet`) — `analyze-form` (reader form → `:op` AST) →
`emit` (AST → Janet form) → Janet `compile`/`eval` — is now **on by default**
in the shipped runtime (`JOLT_INTERPRET=1` opts out). It is a *hybrid*: forms
it can't compile correctly throw `jolt/uncompilable` and fall back to the
interpreter (`loader/eval-toplevel`), so results always match the interpreter.
Validated at parity — conformance 218/218 under both interpret and compile, and
the clojure-test-suite under compile passes 3932 (vs the 3913 interpreter
baseline) across ~4.6k assertions.
- Done so far: var-indirection (globals deref through var cells, so compiled code
is REPL-redefinable); hybrid fallback; compilation of multi-arity / named /
variadic fns and `recur` inside `fn`; map and vector literal compilation
(mode-correct via `make-vec` / `build-map-literal`); resolution that mirrors
the interpreter (current ns → `clojure.core` → Janet-env fallback); and AOT
(`aot.janet`) that marshals a compiled namespace to a Janet bytecode image
against the baked-in runtime dictionary and loads it back.
- Still open — the actual self-hosting: the compiler and most of `clojure.core`
are still Janet. Rewriting them in Clojure (compiled by Jolt) is the remaining
Clojure-in-Clojure work.
## What the host gives us (verified)
Janet already is the backend and the AOT story — we don't need a custom bytecode
emitter:
- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is
Clojure form → correct Janet form → `compile`.
- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled
environment to a **bytecode image** and load it back: this is Phase 4 AOT.
- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the
form layer (we shouldn't need to).
**The catch we verified:** Janet *early-binds* top-level references. Compile
`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns
the old value. So emitting Jolt globals as plain Janet symbols (what the current
compiler largely does) is fundamentally incompatible with REPL redefinition. This
is the single most important design constraint below.
## Prior art
- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is
written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only
~20 special forms are primitive; everything else is macros/functions. Crucially,
compiled call sites go **through Var objects** (a deref), so redefining a var is
visible to existing compiled callers — that's how speed and live redefinition
coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the
var indirection) for speed where you don't need redefinition (used for core in
production). AOT compiles namespaces to `.class` files.
- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus
a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS).
`cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at
~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we
want with Janet as the backend.
- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally
specified* intermediate languages, with autogenerated boilerplate to recur
through unchanged forms and checks that each pass's output matches its grammar.
The lesson for "grow the language as it compiles itself": keep passes small and
IRs explicit so adding a form is local and verifiable.
- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS
(optimization IR) → VM bytecode, with several front-end languages targeting
Tree-IL. The closest analog to "Lisp → bytecode on a VM."
## Assessment: is the current approach the right one?
The overall *shape* is right and matches ClojureScript: front-end (analyze →
emit) with the host VM as the backend, emitting host forms that the host compiles
to bytecode. Two things need to change to reach the goal:
1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref
through the var cell**, not as a Janet symbol. Jolt vars are already cells
(`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly
`((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition
updates the cell's root, so compiled callers see it — exactly Clojure's model.
One indirection per global call; locals and control flow stay direct and fast.
Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition.
2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting
means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is
Clojure. That's the bulk of the work and where the "language builds itself"
payoff lives.
So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for
free), fix global binding, and progressively self-host.
## Recommended architecture
**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small,
named passes rather than one big walker:
1. *read* — reader → forms (already have it).
2. *macroexpand* — fully expand to special forms + calls (the interpreter already
expands; share one expander).
3. *analyze* — forms → AST, resolving locals vs vars and tagging ops.
4. *(optional) optimize* — constant-fold, direct-link hot calls, etc.
5. *emit* — AST → Janet form, with globals as var-cell derefs.
6. *compile* — Janet `compile` → bytecode; `make-image` for AOT.
Make each pass total over the IR so an unhandled node is an explicit gap, not a
silent miss.
**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in
the host before any Clojure can run: the reader; the value/representation layer
(vars, namespaces, symbols, keywords, persistent collections, chars); host
interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur`
evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`,
the macros, and the compiler — is Clojure loaded and (eventually) compiled by the
kernel. Today the kernel is far larger than this; shrinking it is a long game.
**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass
can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`)
for that sub-form instead of erroring. This lets the compiler be incomplete and
still correct (hot paths compile, cold/unsupported paths interpret), lets us grow
coverage incrementally, and de-risks the self-hosting bootstrap.
**Live flexibility.** Vars stay first-class cells; compiled code derefs them;
`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct
linking is opt-in, never the default, so the REPL is always live.
## A staged path
1. **Var-indirection in the emitter** — *done*. Global refs compile as var-cell
derefs, so a compiled `defn` is redefinable at the REPL.
2. **Hybrid fallback + coverage** (`jolt-1bj`) — *done*. Forms the compiler can't
compile throw `jolt/uncompilable` and fall back to the interpreter, so compile
mode is always correct. Covered: multi-arity/named/variadic fns, `recur` in
`fn`, map/vector literals, and resolution matching the interpreter.
Destructuring compiles via the shared `destructure` expander: the `fn`/`let`/
`loop`/`defn` macros desugar to plain-symbol `fn*`/`let*`/`loop*`, so it no
longer falls back — and the primitives reject patterns outright, matching
Clojure (`jolt-f79`).
5. **Compile-by-default + AOT** (`jolt-7j9`) — *done, done out of order*. Once the
hybrid path was validated at parity, compilation was flipped on by default and
AOT images (`aot.janet`) landed. Done before 3–4 because it's the runtime
payoff and only needed the hybrid path to be correct, not self-hosting.
3. **Self-host the compiler** (`jolt-lcn`) — *open*. Rewrite `compiler.janet` as
Clojure (`jolt.compiler`) that Jolt compiles. Now the compiler is part of the
language it compiles.
4. **Shrink the kernel / core-in-Clojure** (`jolt-uqi`) — *open*. Move
`clojure.core` from Janet to Clojure incrementally, each piece compiled by the
previous stage — the language building itself — leaving a minimal Janet kernel.
What remains (3 and 4) is the actual Clojure-in-Clojure rewrite: the largest part
of the work and where the "language builds itself" payoff lives. The correctness
and runtime foundations it needs — redefinable compiled code, an always-correct
hybrid path, compile-by-default, and AOT — are now in place.

91
docs/tools-deps.md Normal file
View file

@ -0,0 +1,91 @@
# deps.edn support — design notes
How Jolt loads pure-Clojure libraries from a `deps.edn`, and why it's built the
way it is. For how to *use* it, see [building-and-deps.md](building-and-deps.md).
Scope, decided up front:
- **git + local deps only** — no Maven/`~/.m2` resolution.
- **pure `clj`/`cljc`** — anything needing the JVM won't load or run; expected.
- **no classpath abstraction** — `require` just needs to find a dep's namespaces;
"the classpath" is an ordered list of source directories.
- **piggyback on jpm** — reuse jpm's git fetch + cache; don't write a package
manager.
- **separate tool** — resolution lives in `jolt-deps`, beside the runtime, the
way `jpm` sits beside `janet`. The `jolt` runtime knows nothing about deps.edn.
## How jpm handles dependencies
jpm's package code (`jpm/pm.janet`) splits into a fetch half and a build half,
and we use only the first:
- **`resolve-bundle`** normalizes a dep spec to `{:url :tag :type :shallow}`,
accepting `:url`/`:repo` + `:tag`/`:sha`/`:commit`/`:ref`. A deps.edn
`{:git/url … :git/sha …}` maps straight onto it.
- **`download-bundle url :git tag shallow`** clones into a content-addressed cache
(`<modpath>/.cache/git_<tag>_<sanitized-url>`) and returns the path —
`git init` + `remote add` + fetch + reset, plus submodules. No build step.
- **`bundle-install`** is the half we skip: it then runs `project.janet` build
rules, which a Clojure lib doesn't have. It's cleanly separable from the clone.
So jpm gives us git resolution and a cache for free; calling `download-bundle`
needs `jpm/config/load-default` first (it sets `gitpath` and the cache dyns).
## How it works
`src/jolt/deps.janet` reads `deps.edn` (Janet parses it directly — EDN and Janet
syntax overlap for the `:deps`/`:paths` subset), then walks `:deps`:
- `:git/url` (+ `:git/sha` or `:git/tag`) → `resolve-bundle` + `download-bundle`
into `jpm_tree/.cache`;
- `:local/root` → the path as-is;
- `:mvn/*` and anything else → ignored.
Each resolved dependency contributes its own `:paths` (default `["src"]`) as
source roots, and we recurse into its `deps.edn` for transitive deps. The result
is a de-duplicated, ordered list of directories. `resolve-deps-cached` memoizes
that list in the tree keyed on a hash of `deps.edn`, so an unchanged file doesn't
re-fetch. jpm is loaded lazily (`require`, not `import`) so it's pulled in only
when resolving — never embedded in a built binary.
The loader (`evaluator.janet/find-ns-file`) resolves a namespace by searching the
context's `:source-paths` in order (the stdlib `src/jolt` first), trying `<ns>.clj`
then `<ns>.cljc`. Extra roots come from `JOLT_PATH` or `init`'s `:paths` option.
`jolt-deps` (`src/jolt/deps_cli.janet`, its own `declare-executable`) ties it
together: it resolves the roots and runs the `jolt` binary with them on
`JOLT_PATH`. The runtime's only dependency interface is that env var.
`jolt uberscript` bundles a namespace and everything it requires into one
standalone `.clj`. It requires the entry namespace and uses the order in which
the loader finishes loading files — a dependency finishes before the file that
required it, so the order is topological — then concatenates that source. The
baked-in stdlib is excluded (it's part of the runtime, not bundled).
Gotcha worth remembering: the `jolt` CLI's context is built into its image at
build time, so `JOLT_PATH` is applied at runtime in `main`, not in `init` (whose
env read would be frozen at build).
## Limitations
- Pure `clj`/`cljc` only — JVM interop, host classes, and unimplemented
`clojure.core` corners fail. Coverage is per-function: a namespace can load with
most functions working and a few not.
- Source only; compiled `.class` files in a git dep are ignored.
- git `:git/sha` must be a full SHA (`git fetch` can't resolve a short one).
## Conformance
`test/integration/deps-conformance-test.janet` resolves a few real pure-`cljc`
git libraries and reports whether their namespaces load and a sample call works.
It's network-gated behind `JOLT_CONFORMANCE=1` so CI stays offline. Use it to
check a library against the current interpreter, and to drive fixes for whatever
gap a failure points at (the same loop as the clojure-test-suite battery). A
library fails when it relies on something Jolt doesn't provide — JVM interop, or
a regex feature like Unicode property classes (`\p{…}`).
## Not yet
- **Compiling deps into a binary image.** `uberscript` already produces a
standalone `.clj`; baking a project's dependencies directly into a custom
executable image is a heavier variant that isn't implemented.