diff --git a/doc/self-hosting-architecture.md b/doc/self-hosting-architecture.md new file mode 100644 index 0000000..e208b87 --- /dev/null +++ b/doc/self-hosting-architecture.md @@ -0,0 +1,138 @@ +# Self-hosting architecture: portable jolt-core over a host runtime + +Design for splitting Jolt into a **portable Clojure-in-Clojure core** and a +**host runtime** (Janet today, another runtime tomorrow), so the language is +truly self-hosted and `jolt-core` can be lifted out and re-hosted. + +This is the design that must be right *before* writing the compiler in Clojure — +see [[self-hosting-compiler]] for the staged plan it plugs into. + +## What "truly self-hosted + portable" requires + +Two independent properties: + +1. **Self-hosted** — the compiler and most of `clojure.core` are written in + Clojure and compiled by Jolt itself. +2. **Portable** — that Clojure code (`jolt-core`) depends only on a small, + explicit **host contract**, never on Janet directly. Re-hosting means + implementing the contract for a new runtime; `jolt-core` is reused verbatim. + +The enemy is `jolt-core` calling `janet/tuple`, `make-vec`, `ns-find`, etc. +directly — that welds it to Janet. Every host dependency must go through the +contract. + +## Prior art (the seam everyone uses) + +- **Clojure (JVM).** `clojure.lang.*` (Java) is the host: `RT`/`Numbers` runtime + helpers, the `Compiler` (form → JVM bytecode), persistent data structures, + `Var`/`Namespace`, the reader. `clojure/core.clj` is the language, in Clojure. + Seam: ~20 primitive special forms + `RT` static methods. Everything else is + Clojure. +- **ClojureScript (self-hosted).** Two portable passes — `cljs.analyzer` + (form → AST **as data**, reading a **compiler-state map** of + namespaces/defs/macros, *not* host objects) and `cljs.compiler` (AST → JS, the + host-specific back end). `cljs.core` is Clojure compiled to JS. Platform splits + live in `.cljc` reader conditionals. This is the closest model to what we want: + **the analyzer is host-agnostic; only the back end and the runtime are + host-specific.** +- **Nanopass / Guile Tree-IL.** A high-level IR is the portability seam; multiple + back ends consume it. +- **ClojureCLR / ClojureDart / jank.** Same shape every time: portable analyzer + + host back end + host runtime. + +The invariant across all of them: **the IR (analyzer output) and a small runtime +protocol are the contract; the front end is portable, the back end and runtime +are per-host.** + +## Decisions (locked) + +- **Seam = a minimal host protocol.** `jolt-core` calls a small documented set of + host fns (in ns `jolt.host`): `resolve-sym`, `macro?`, `macroexpand-1`, + `current-ns`, `intern!`, plus the `RT` primitives. Each host provides `jolt.host` + (+ RT). Re-hosting = reimplement that handful of fns. The protocol *is* the + boundary; `jolt-core` never touches Janet directly. +- **Physical split now.** Portable Clojure lives under `jolt-core/` (a new source + root, embedded into the binary like the rest of the stdlib); host Janet code for + the new pipeline under `host/janet/`. Legacy host modules under `src/jolt/*.janet` + are the existing Janet host and get relocated under `host/janet/` in a later + mechanical pass (tracked) — not moved big-bang now, to keep the suite green. + +## The Jolt split + +``` +jolt-core/ PORTABLE Clojure — no Janet. Depends only on the contract. + ir the IR spec (data shapes the analyzer emits) + analyzer form -> IR (macroexpands; resolves via host protocol) + macros when/cond/->/defn/... (the macro library, in Clojure) + core clojure.core fns expressible in Clojure, over RT primitives + +host/janet/ THE HOST — Janet. Implements the contract. + reader text -> jolt forms + rt data structures + RT primitive fns (cons/first/+/get/apply…) + backend IR -> Janet forms -> Janet compile -> bytecode (the emitter) + cenv the compile-time host protocol impl (resolve/macro?/intern) + bootstrap load jolt-core, wire analyzer+backend into the loader + interop janet.* bridge +``` + +Two contracts cross the seam: + +### 1. The IR (analyzer → back end) +The existing `:op`-tagged AST, made **host-neutral**: +- `{:op :const :val v}`, `:if`, `:do`, `:let`, `:fn` (arities), `:invoke`, + `:vector`/`:map`/`:set`, `:quote`, `:throw`/`:try`, `:loop`/`:recur`. +- **Globals reference vars by NAME, not by host cell:** + `{:op :var :ns "clojure.core" :name "map"}`. (compiler.janet today embeds the + Janet var cell as a constant — that's a host leak and breaks AOT. Name-based + refs are both portable and AOT-friendly; the back end resolves the cell.) +- No embedded host function values. Calls to runtime primitives are + `{:op :rt :name "cons"}` resolved by the back end to the host's RT fn. + +### 2. The host contract (two protocols) +- **Compile-time (`cenv`)** — what the analyzer needs from the host while + analyzing: `(current-ns)`, `(resolve-sym sym) -> {:kind :var|:macro|:local|:special|:host, :ns, :name}`, + `(macroexpand-1 form)`, `(intern! ns sym meta)`. The analyzer calls only these; + it never touches Janet ns/var tables. (CLJS keeps this as pure data; we use a + small protocol — a minimal, documented boundary — because Jolt already has live + ns/var objects. The protocol *is* the seam.) +- **Runtime (`RT`)** — the primitive fns emitted code and `jolt-core` call by + stable name: arithmetic/compare, `cons/first/rest/seq/conj/get/assoc/count`, + `apply`, `=`, vector/map/set constructors, var deref/bind, keyword/symbol + construction. The back end maps each to the host (on Janet, mostly the existing + `core-*`). To re-host, implement this set. + +## Why name-based vars (not embedded cells) + +`compiler.janet` compiles a global ref to a closure over the Janet var cell. That +(a) is a Janet value baked into the IR — not portable, and (b) can't be marshaled +for AOT without the runtime-dict trick. Compiling instead to *resolve var by +(ns,name) at call time* through an RT primitive keeps redefinition live, makes the +IR host-neutral, and makes images trivially portable. The per-call lookup is the +cost; it can be cached/direct-linked later as an opt-in optimization. + +## Bootstrap & staging (keeps the suite green throughout) + +`compiler.janet` stays as the **bootstrap back end** until the Clojure pipeline is +proven. Order: + +1. **Freeze the IR** spec and refactor `compiler.janet`'s emit to consume + name-based `:var` (no behavior change; bootstrap still works). +2. **Define the host contract** (`cenv` + `RT`) and implement it on Janet, + exposed under a stable namespace the Clojure core can call. +3. **Write `jolt.analyzer` in Clojure** producing IR, against `cenv`. Diff its IR + against the Janet analyzer on the conformance corpus until identical. +4. **Janet back end consumes IR** from the Clojure analyzer; wire into the loader + behind a flag. Validate at parity (dual-mode conformance + clojure-test-suite). +5. **Flip** the loader to the Clojure analyzer + Janet back end; `compiler.janet` + shrinks to the back end only. +6. **Move `clojure.core`** macros then fns into `jolt-core` incrementally, each + compiled by the prior stage, isolating host bits behind `RT`. + +Guards at every step: the dual-mode conformance harness (interpret vs compile) +and the clojure-test-suite baseline. + +## The portability test + +When done, re-hosting Jolt to runtime X means writing only: `host/X/{reader, rt, +backend, cenv, bootstrap}`. `jolt-core/{ir, analyzer, macros, core}` is reused +unchanged. That is the concrete bar for "truly self-hosted and portable." diff --git a/jolt-core/jolt/ir.clj b/jolt-core/jolt/ir.clj new file mode 100644 index 0000000..01fd97a --- /dev/null +++ b/jolt-core/jolt/ir.clj @@ -0,0 +1,42 @@ +(ns jolt.ir + "Host-neutral intermediate representation for the Jolt compiler. + + The analyzer (jolt.analyzer) produces IR; a host back end consumes it. IR nodes + are plain maps tagged with :op — no host values embedded. Globals reference vars + by name (:ns/:name), never by a host var cell, so the IR is portable and + AOT-safe. This namespace is pure Clojure (portable jolt-core): it depends on + nothing host-specific.") + +;; Node constructors. Kept as data so any back end can pattern-match on :op. + +(defn const [v] {:op :const :val v}) + +(defn local [name] {:op :local :name name}) + +;; A global var reference, by name. The back end resolves it to a host var. +(defn var-ref [ns name] {:op :var :ns ns :name name}) + +;; A runtime primitive (cons, +, get, apply, …) the back end maps to the host RT. +(defn rt [name] {:op :rt :name name}) + +(defn if-node [test then else] {:op :if :test test :then then :else else}) + +(defn do-node [statements ret] {:op :do :statements statements :ret ret}) + +(defn invoke [f args] {:op :invoke :fn f :args args}) + +(defn def-node [ns name init] {:op :def :ns ns :name name :init init}) + +(defn let-node [bindings body] {:op :let :bindings bindings :body body}) + +;; A fn is one or more arities. Each arity: {:params [..] :rest name|nil :body ir}. +(defn fn-node [name arities] {:op :fn :name name :arities arities}) + +(defn vector-node [items] {:op :vector :items items}) +(defn map-node [pairs] {:op :map :pairs pairs}) +(defn set-node [items] {:op :set :items items}) + +(defn quote-node [form] {:op :quote :form form}) +(defn throw-node [expr] {:op :throw :expr expr}) + +(defn op [node] (:op node)) diff --git a/src/jolt/stdlib_embed.janet b/src/jolt/stdlib_embed.janet index e63c239..6dd5eb6 100644 --- a/src/jolt/stdlib_embed.janet +++ b/src/jolt/stdlib_embed.janet @@ -30,4 +30,6 @@ (let [acc @{}] (collect "src/jolt/clojure" "clojure/" acc) (collect "src/jolt/jolt" "jolt/" acc) + # Portable Clojure core (analyzer/IR/…): jolt-core/jolt/foo.clj -> jolt.foo + (collect "jolt-core" "" acc) acc)) diff --git a/src/jolt/types.janet b/src/jolt/types.janet index 7e07e20..ad6bd51 100644 --- a/src/jolt/types.janet +++ b/src/jolt/types.janet @@ -370,8 +370,9 @@ :current-ns "user" :compile? compile? # Ordered roots searched (after the stdlib) to resolve a namespace - # to a .clj/.cljc file. deps.edn resolution appends dep src dirs. - :source-paths @["src/jolt"] + # to a .clj/.cljc file. jolt-core holds the portable Clojure layer + # (analyzer/IR/core); deps.edn resolution appends dep src dirs. + :source-paths @["jolt-core" "src/jolt"] :compiled-cache @{} :type-registry @{} :data-readers (let [dr @{}]