Stage2 compile only (#12)

* core: compile macro expanders via staged bootstrap (Stage 2 Task 1)

Macros are now compiled, not interpreted, by steady state — matching
Clojure (macros are ordinary compiled fns the Java seed compiles for
clojure.core) and ClojureScript (macros compiled, invoked at compile
time). Neither reference keeps an interpreted-closure fallback.

The early macros (00-syntax) are defined while the self-hosted analyzer
is still being bootstrapped (it builds lazily after the overlay loads),
so macro-compile-hook returns nil and they get an interpreted closure.
The bootstrap compiler.janet can't substitute (it punts on syntax-quote,
which nearly every expander uses).

Fix = staged bootstrap, the same pattern as the compiler fixpoint:
defmacro stashes the expander source on the var (:macro-src) plus a
:macro-uses-env flag; once the overlay + analyzer are fully built,
backend/recompile-macros! (via ensure-macros-compiled! at the end of
load-core-overlay!) compiles each stashed expander through the now-live
analyzer and rebinds the var, marking :macro-compiled. Idempotent;
&env/&form macros keep the interpreted closure (the compiled fn* has no
such params). The interpreter is now a build-time crutch, gone by
steady state.

Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring
(which the analyzer punts on) to a plain rest param + (first rest), so
all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79.

47/47 overlay macros compiled, 0 interpreted; user macros compile
immediately post-init. Gate green: conformance 267x3, fallback-zero
31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap,
lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap,
features 78/78, all unit+spec, core-bench neutral.

* core: wrap macro expanders in fn (not fn*) so destructured arglists compile

Follow-up to the staged macro-compile change. The macro-recompile pass
wrapped each expander in the raw fn* primitive, which punts on a
destructuring rest param — so if-not/if-let/if-some/assert (using the
canonical '& [else]') couldn't compile and had been rewritten to plain
rest params as a workaround.

Root cause: fn* is the primitive; the fn MACRO is what desugars
destructuring (rest, map, nested) into the body before lowering. Wrapping
expanders in fn instead of fn* compiles any destructured macro arglist
uniformly, so the workaround is unnecessary — reverted those 4 macros to
the canonical '& [else]' forms.

Net: rest-destructuring is fully compiled for all normal code (fn/defn/
let/macro params). Only the hand-written raw fn* primitive still punts
(jolt-f79, downgraded to P4 — falls back to interpreter, still correct).

47/47 overlay macros compiled. Gate green: conformance 267x3,
fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host,
staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67,
sci-bootstrap, features, all unit+spec.

* core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79)

jolt-f79 asked to compile destructuring in fn* rest params. Checking
against Clojure inverts the premise: Clojure's fn* REJECTS destructuring
at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding
form, expected symbol'). So the self-hosted analyzer was already correct
— fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn
MACROS desugar destructuring. The real defect was the interpreter
leniently destructuring raw fn*, and defn emitting raw fn* to rely on it.

Changes:
- evaluator: fn*/let*/loop* now reject non-symbol binding forms with
  Clojure's exact messages (require-symbol-params/plain-sym?), so the
  interpreter agrees with the analyzer + Clojure.
- 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring
  params desugar; unnamed, so self-recursion still resolves via the var.
- 00-syntax: completing that exposed a real gap — the overlay destructure
  fn didn't handle kwargs (a map pattern bound against a fn's sequential
  rest); it had only worked via the interpreter's destructure-bind. Added
  the seq->map coercion to the map? branch (sequential: 1 map elt => that
  map, else apply hash-map), matching destructure-bind so interpret ==
  compile.

Net: fn*/let*/loop* are plain-symbol primitives across interpreter,
analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro
params, incl kwargs & {:keys}) compiles through the macros with no
interpreter fallback. Regression spec added.

Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44,
clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all
unit+spec (destructuring 50/50).

* docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79)

grammar.ebnf: the destructuring note already attributed patterns to the
binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES
they desugar to take plain symbols only (a non-symbol binding errors, as
in Clojure).

self-hosting-compiler.md: the 'compile destructuring via a shared
destructure expander instead of falling back' item is done (and its
jolt-7dl ref was stale) — destructuring now compiles through the
fn/let/loop/defn macros' desugaring; the primitives reject patterns.

* core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy

First slice of moving stateful forms onto the compile path (jolt-eaa).
- loader stateful-head?: drop syntax-quote (the analyzer's `handled` set
  already compiles it; routing it to the interpreter was redundant).
- host_iface special-names: drop definterface/extend/proxy (stub macros
  expanding to def/nil — their expansions compile once unpunted).
- letfn stays interpreted: its let* expansion needs letrec semantics
  (mutual recursion between the fns), which sequential compiled let* lacks
  — a later tier.

Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + protocol/multimethod/macro specs.

---------

Co-authored-by: Yogthos <yogthos@gmail.com>

src/jolt/api.janet

@@ -14,14 +14,21 @@
 (import ./stdlib_embed :as stdlib-embed)
 (import ./host_iface :as host)
 
-# A defmacro expander compiles to a native fn (built as (fn* args body...) and run
-# through the self-hosted pipeline) so macro expansion is compiled, zero runtime
-# cost — instead of an interpreted closure. Returns nil (interpreted fallback) when
-# the analyzer isn't built yet or the body isn't compilable.
+# A defmacro expander compiles to a native fn (built as (fn args body...) and run
+# through the self-hosted pipeline) so macro expansion is COMPILED code, zero runtime
+# cost — instead of an interpreted closure, mirroring Clojure (macros are ordinary
+# compiled fns). Wrapped in the `fn` MACRO (not the `fn*` primitive) so a destructured
+# macro arglist — `[a & [b]]`, `[& {:keys [x]}]`, nested — desugars before lowering;
+# raw fn* punts on a destructuring rest param. Returns nil when the analyzer isn't
+# built yet (the early macros, expanded WHILE the analyzer is being bootstrapped) or
+# the body isn't compilable; in that case defmacro keeps an interpreted closure, and
+# backend/recompile-macros! replaces it with a compiled expander once the analyzer
+# comes alive (staged bootstrap — the interpreter is a build-time crutch, gone by
+# steady state).
 (set macro-compile-hook
   (fn [ctx args-form body]
     (backend/try-compile-fn ctx
-      (array/concat @[{:jolt/type :symbol :ns nil :name "fn*"} args-form] body))))
+      (array/concat @[{:jolt/type :symbol :ns nil :name "fn"} args-form] body))))
 
 (defn normalize-pvecs
   "Deep-convert any sequential (pvec/tuple/array) to a Janet tuple. Test helper
@@ -92,7 +99,12 @@
       # half-loaded core (which would forward-ref the missing kernel fns to nil).
       (when (tier :kernel) (put env :kernel-ready? true))))
   (put env :direct-linking? user-dl)
-  (ctx-set-current-ns ctx saved))
+  (ctx-set-current-ns ctx saved)
+  # Staged bootstrap: the early macros (00-syntax) were defined while the analyzer
+  # was still being built, so their expanders are interpreted closures. Now that the
+  # full overlay + analyzer are in place, recompile those expanders to native code —
+  # by steady state no macro expansion runs interpreted (no-op in interpreter mode).
+  (backend/ensure-macros-compiled! ctx))
 
 (defn init
   "Create a new Jolt evaluation context.

src/jolt/backend.janet

@@ -372,3 +372,49 @@
     (when (compiled 0)
       (def r (protect (eval (compiled 1) (comp/ctx-janet-env ctx))))
       (when (r 0) (r 1)))))
+
+# Wrap expanders in the `fn` MACRO, not the `fn*` primitive: `fn` desugars a
+# destructured macro arglist (`[a & [b]]`, `[& {:keys [x]}]`) before lowering,
+# whereas raw fn* punts on a destructuring rest param.
+(def- fn-sym {:jolt/type :symbol :ns nil :name "fn"})
+
+(defn recompile-macros!
+  "Staged-bootstrap second pass: once the self-hosted analyzer is alive, replace
+  every interpreted macro expander with a COMPILED one. The early macros (00-syntax
+  etc.) are defined WHILE the analyzer is still being bootstrapped, so their
+  expanders can't compile yet (the analyzer they'd compile through doesn't exist) —
+  defmacro gives them an interpreted closure as a build-time crutch and stashes the
+  source on the var (:macro-src). This pass compiles that source through the now-live
+  analyzer and rebinds the var, so by steady state no macro expansion is interpreted
+  — mirroring how a self-hosting compiler recompiles its seed once it can.
+
+  Idempotent: a var compiled once is marked :macro-compiled and skipped (so the
+  refer of a core macro into another ns, or a later rebuild, costs nothing). A macro
+  whose body uses &env/&form keeps its interpreted closure (the compiled fn* has no
+  such params). Returns the number of expanders compiled this pass."
+  [ctx]
+  (var n 0)
+  (each ns (all-ns ctx)
+    (each v (ns :mappings)
+      (when (and (var? v) (var-macro? v)
+                 (v :macro-src) (not (v :macro-compiled))
+                 (not (v :macro-uses-env)))
+        (def [args-form body] (v :macro-src))
+        (def compiled
+          (try-compile-fn ctx (array/concat @[fn-sym args-form] body)))
+        (when compiled
+          (bind-root v compiled)
+          (put v :macro-compiled true)
+          (++ n)))))
+  n)
+
+(defn ensure-macros-compiled!
+  "Called once the overlay is fully loaded (api/load-core-overlay!): in compile
+  mode, ensure the analyzer is built, then run the staged macro-recompile pass so
+  the early (interpreted-during-bootstrap) macro expanders become compiled. No-op
+  in interpreter mode (no analyzer, macros stay interpreted by design) and cheap to
+  call again (recompile-macros! skips already-compiled vars)."
+  [ctx]
+  (when (get (ctx :env) :compile?)
+    (ensure-analyzer ctx)
+    (when (analyzer-built? ctx) (recompile-macros! ctx))))

src/jolt/evaluator.janet

@@ -517,6 +517,21 @@
         (do (array/push fixed a) (+= i 1)))))
   {:fixed (tuple/slice (tuple ;fixed)) :rest rest-pat})
 
+(defn- plain-sym? [p] (and (struct? p) (= :symbol (p :jolt/type))))
+
+(defn- require-symbol-params
+  "fn* is a primitive: its params must be plain symbols. The fn/defn MACROS desugar
+  destructuring into plain params + a body let before emitting fn*, so fn* never
+  legitimately sees a pattern — matching Clojure, where (fn* [[a b]] ...) is the
+  compile error 'fn params must be Symbols'. Enforcing it here keeps the interpreter
+  consistent with the self-hosted analyzer (which also requires plain fn* params)
+  and with Clojure, instead of leniently destructuring a form Clojure rejects."
+  [param-info]
+  (each p (param-info :fixed)
+    (unless (plain-sym? p) (error "fn params must be Symbols")))
+  (let [r (param-info :rest)]
+    (when (and r (not (plain-sym? r))) (error "fn params must be Symbols"))))
+
 (defn- d-get
   "Look up key k in a map-like value (phm/struct/table/nil)."
   [m k]
@@ -805,6 +820,14 @@
                        ns (ctx-find-ns ctx ns-name)]
                    (def v (ns-intern ns (name-sym :name) macro-fn))
                    (put v :macro true)
+                   # Stash the expander source so backend/recompile-macros! can
+                   # compile it once the analyzer is alive (staged bootstrap): a
+                   # macro defined WHILE the analyzer is still being built gets an
+                   # interpreted closure now, a compiled expander later. uses-env
+                   # macros stay interpreted (the compiled fn* has no &env/&form).
+                   (put v :macro-src @[args-form body])
+                   (put v :macro-uses-env uses-env)
+                   (when compiled-fn (put v :macro-compiled true))
                    # A (re)defined macro invalidates any cached expansions.
                    (table/clear macro-cache)
                    (var-get v)))
@@ -917,6 +940,7 @@
                  (let [args-form (in pair 0)
                        body (tuple/slice pair 1)
                        param-info (parse-params args-form)
+                       _ (require-symbol-params param-info)
                        fixed-pats (param-info :fixed)
                        rest-pat (param-info :rest)
                        n-fixed (length fixed-pats)
@@ -954,6 +978,7 @@
              (let [args-form (in form 1)
                    body (tuple/slice form 2)
                    param-info (parse-params args-form)
+                   _ (require-symbol-params param-info)
                    fixed-pats (param-info :fixed)
                    rest-pat (param-info :rest)
                    defining-ns (ctx-current-ns ctx)]
@@ -986,6 +1011,9 @@
               (let [len (length bind-vec)]
                 (while (< i len)
                   (let [pat (bind-vec i)]
+                    # let* is a primitive (the let macro desugars destructuring);
+                    # its binding names must be plain symbols, as in Clojure.
+                    (unless (plain-sym? pat) (error "Bad binding form, expected symbol"))
                     (def val (eval-form ctx new-bindings (bind-vec (+ i 1))))
                     (destructure-bind ctx new-bindings pat val)
                     (+= i 2))))
@@ -999,8 +1027,10 @@
                   patterns @[]]
               (var i 0)
               (while (< i (length bind-vec))
+                # loop* is a primitive (the loop macro desugars destructuring);
+                # its binding names must be plain symbols, as in Clojure.
+                (unless (plain-sym? (bind-vec i)) (error "Bad binding form, expected symbol"))
                 (array/push init-vals (eval-form ctx bindings (bind-vec (+ i 1))))
-                # keep the binding form (symbol OR destructuring pattern)
                 (array/push patterns (bind-vec i))
                 (+= i 2))
               (var loop-fn nil)

src/jolt/host_iface.janet

@@ -81,8 +81,10 @@
              "create-ns" "remove-ns" "find-ns" "all-ns" "the-ns" "resolve"
              "ns-resolve" "ns-aliases" "ns-imports" "ns-interns"
              "read-string" "macroexpand-1" "defonce" "ns" "in-ns" "require"
-             "import" "use" "refer" "defrecord" "defprotocol" "definterface"
-             "reify" "proxy" "extend-type" "extend-protocol" "extend" "gen-class"
+             "import" "use" "refer" "defrecord" "defprotocol"
+             "reify" "extend-type" "extend-protocol" "gen-class"
+             # letfn stays: its let* expansion needs letrec semantics (mutual
+             # recursion between the fns), which compiled sequential let* lacks.
              "monitor-enter" "monitor-exit" "binding" "letfn"]
       (put t n true))
     t))

src/jolt/loader.janet

@@ -9,12 +9,14 @@
 # Stateful / context-modifying forms always interpret: they mutate the context
 # (namespaces, macros, types, multimethods, dynamic vars, …) in ways the compiler
 # doesn't model. Kept here so the compile/interpret routing lives in one place,
-# used by both load-ns and the public eval-one.
+# used by both load-ns and the public eval-one. Shrinking toward the frozen
+# host-coupled set (Stage 2 jolt-eaa): forms move off this list as they gain a
+# compile path; syntax-quote already compiles via the analyzer's `handled` set.
 (defn- stateful-head? [head-name]
   (or (= head-name "defmacro") (= head-name "ns")
       (= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod")
       (= head-name "require") (= head-name "in-ns")
-      (= head-name "syntax-quote") (= head-name "set!")
+      (= head-name "set!")
       (= head-name "var") (= head-name ".") (= head-name "new")
       (= head-name "eval")))