jolt/docs/self-hosting-architecture.md

Self-hosting architecture: portable jolt-core over a host runtime

Design for splitting Jolt into a portable Clojure-in-Clojure core and a host runtime (Janet today, another runtime tomorrow), so the language is truly self-hosted and jolt-core can be lifted out and re-hosted.

This is the design that must be right before writing the compiler in Clojure — see self-hosting-compiler for the staged plan it plugs into.

What "truly self-hosted + portable" requires

Two independent properties:

1. Self-hosted — the compiler and most of clojure.core are written in Clojure and compiled by Jolt itself.
2. Portable — that Clojure code (jolt-core) depends only on a small, explicit host contract, never on Janet directly. Re-hosting means implementing the contract for a new runtime; jolt-core is reused verbatim.

The enemy is jolt-core calling janet/tuple, make-vec, ns-find, etc. directly — that welds it to Janet. Every host dependency must go through the contract.

Prior art (the seam everyone uses)

• Clojure (JVM). clojure.lang.* (Java) is the host: RT/Numbers runtime helpers, the Compiler (form → JVM bytecode), persistent data structures, Var/Namespace, the reader. clojure/core.clj is the language, in Clojure. Seam: ~20 primitive special forms + RT static methods. Everything else is Clojure.
• ClojureScript (self-hosted). Two portable passes — cljs.analyzer (form → AST as data, reading a compiler-state map of namespaces/defs/macros, not host objects) and cljs.compiler (AST → JS, the host-specific back end). cljs.core is Clojure compiled to JS. Platform splits live in .cljc reader conditionals. This is the closest model to what we want: the analyzer is host-agnostic; only the back end and the runtime are host-specific.
• Nanopass / Guile Tree-IL. A high-level IR is the portability seam; multiple back ends consume it.
• ClojureCLR / ClojureDart / jank. Same shape every time: portable analyzer + host back end + host runtime.

The invariant across all of them: the IR (analyzer output) and a small runtime protocol are the contract; the front end is portable, the back end and runtime are per-host.

Decisions (locked)

• Seam = a minimal host protocol. jolt-core calls a small documented set of host fns (in ns jolt.host): resolve-sym, macro?, macroexpand-1, current-ns, intern!, plus the RT primitives. Each host provides jolt.host (+ RT). Re-hosting = reimplement that handful of fns. The protocol is the boundary; jolt-core never touches Janet directly.
• Physical split now. Portable Clojure lives under jolt-core/ (a new source root, embedded into the binary like the rest of the stdlib); host Janet code for the new pipeline under host/janet/. Legacy host modules under src/jolt/*.janet are the existing Janet host and get relocated under host/janet/ in a later mechanical pass (tracked) — not moved big-bang now, to keep the suite green.

The Jolt split

jolt-core/            PORTABLE Clojure — no Janet. Depends only on the contract.
  ir                  the IR spec (data shapes the analyzer emits)
  analyzer            form -> IR        (macroexpands; resolves via host protocol)
  macros              when/cond/->/defn/... (the macro library, in Clojure)
  core                clojure.core fns expressible in Clojure, over RT primitives

host/janet/           THE HOST — Janet. Implements the contract.
  reader              text -> jolt forms
  rt                  data structures + RT primitive fns (cons/first/+/get/apply…)
  backend             IR -> Janet forms -> Janet compile -> bytecode  (the emitter)
  cenv                the compile-time host protocol impl (resolve/macro?/intern)
  bootstrap           load jolt-core, wire analyzer+backend into the loader
  interop             janet.* bridge

Two contracts cross the seam:

1. The IR (analyzer → back end)

The existing :op-tagged AST, made host-neutral:

• {:op :const :val v}, :if, :do, :let, :fn (arities), :invoke, :vector/:map/:set, :quote, :throw/:try, :loop/:recur.
• Globals reference vars by NAME, not by host cell: {:op :var :ns "clojure.core" :name "map"}. (compiler.janet today embeds the Janet var cell as a constant — that's a host leak and breaks AOT. Name-based refs are both portable and AOT-friendly; the back end resolves the cell.)
• No embedded host function values. Calls to runtime primitives are {:op :rt :name "cons"} resolved by the back end to the host's RT fn.

2. The host contract (two protocols)

• Compile-time (cenv) — what the analyzer needs from the host while analyzing: (current-ns), (resolve-sym sym) -> {:kind :var|:macro|:local|:special|:host, :ns, :name}, (macroexpand-1 form), (intern! ns sym meta). The analyzer calls only these; it never touches Janet ns/var tables. (CLJS keeps this as pure data; we use a small protocol — a minimal, documented boundary — because Jolt already has live ns/var objects. The protocol is the seam.)
• Runtime (RT) — the primitive fns emitted code and jolt-core call by stable name: arithmetic/compare, cons/first/rest/seq/conj/get/assoc/count, apply, =, vector/map/set constructors, var deref/bind, keyword/symbol construction. The back end maps each to the host (on Janet, mostly the existing core-*). To re-host, implement this set.

Why name-based vars (not embedded cells)

compiler.janet compiles a global ref to a closure over the Janet var cell. That (a) is a Janet value baked into the IR — not portable, and (b) can't be marshaled for AOT without the runtime-dict trick. Compiling instead to resolve var by (ns,name) at call time through an RT primitive keeps redefinition live, makes the IR host-neutral, and makes images trivially portable. The per-call lookup is the cost; it can be cached/direct-linked later as an opt-in optimization.

Bootstrap & staging (keeps the suite green throughout)

compiler.janet stays as the bootstrap back end until the Clojure pipeline is proven. Order:

1. Freeze the IR spec and refactor compiler.janet's emit to consume name-based :var (no behavior change; bootstrap still works).
2. Define the host contract (cenv + RT) and implement it on Janet, exposed under a stable namespace the Clojure core can call.
3. Write jolt.analyzer in Clojure producing IR, against cenv. Diff its IR against the Janet analyzer on the conformance corpus until identical.
4. Janet back end consumes IR from the Clojure analyzer; wire into the loader behind a flag. Validate at parity (dual-mode conformance + clojure-test-suite).
5. Flip the loader to the Clojure analyzer + Janet back end; compiler.janet shrinks to the back end only.
6. Move clojure.core macros then fns into jolt-core incrementally, each compiled by the prior stage, isolating host bits behind RT.

Guards at every step: the dual-mode conformance harness (interpret vs compile) and the clojure-test-suite baseline.

The portability test

When done, re-hosting Jolt to runtime X means writing only: host/X/{reader, rt, backend, cenv, bootstrap}. jolt-core/{ir, analyzer, macros, core} is reused unchanged. That is the concrete bar for "truly self-hosted and portable."