Stage2 compile only (#12)

* core: compile macro expanders via staged bootstrap (Stage 2 Task 1)

Macros are now compiled, not interpreted, by steady state — matching
Clojure (macros are ordinary compiled fns the Java seed compiles for
clojure.core) and ClojureScript (macros compiled, invoked at compile
time). Neither reference keeps an interpreted-closure fallback.

The early macros (00-syntax) are defined while the self-hosted analyzer
is still being bootstrapped (it builds lazily after the overlay loads),
so macro-compile-hook returns nil and they get an interpreted closure.
The bootstrap compiler.janet can't substitute (it punts on syntax-quote,
which nearly every expander uses).

Fix = staged bootstrap, the same pattern as the compiler fixpoint:
defmacro stashes the expander source on the var (:macro-src) plus a
:macro-uses-env flag; once the overlay + analyzer are fully built,
backend/recompile-macros! (via ensure-macros-compiled! at the end of
load-core-overlay!) compiles each stashed expander through the now-live
analyzer and rebinds the var, marking :macro-compiled. Idempotent;
&env/&form macros keep the interpreted closure (the compiled fn* has no
such params). The interpreter is now a build-time crutch, gone by
steady state.

Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring
(which the analyzer punts on) to a plain rest param + (first rest), so
all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79.

47/47 overlay macros compiled, 0 interpreted; user macros compile
immediately post-init. Gate green: conformance 267x3, fallback-zero
31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap,
lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap,
features 78/78, all unit+spec, core-bench neutral.

* core: wrap macro expanders in fn (not fn*) so destructured arglists compile

Follow-up to the staged macro-compile change. The macro-recompile pass
wrapped each expander in the raw fn* primitive, which punts on a
destructuring rest param — so if-not/if-let/if-some/assert (using the
canonical '& [else]') couldn't compile and had been rewritten to plain
rest params as a workaround.

Root cause: fn* is the primitive; the fn MACRO is what desugars
destructuring (rest, map, nested) into the body before lowering. Wrapping
expanders in fn instead of fn* compiles any destructured macro arglist
uniformly, so the workaround is unnecessary — reverted those 4 macros to
the canonical '& [else]' forms.

Net: rest-destructuring is fully compiled for all normal code (fn/defn/
let/macro params). Only the hand-written raw fn* primitive still punts
(jolt-f79, downgraded to P4 — falls back to interpreter, still correct).

47/47 overlay macros compiled. Gate green: conformance 267x3,
fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host,
staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67,
sci-bootstrap, features, all unit+spec.

* core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79)

jolt-f79 asked to compile destructuring in fn* rest params. Checking
against Clojure inverts the premise: Clojure's fn* REJECTS destructuring
at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding
form, expected symbol'). So the self-hosted analyzer was already correct
— fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn
MACROS desugar destructuring. The real defect was the interpreter
leniently destructuring raw fn*, and defn emitting raw fn* to rely on it.

Changes:
- evaluator: fn*/let*/loop* now reject non-symbol binding forms with
  Clojure's exact messages (require-symbol-params/plain-sym?), so the
  interpreter agrees with the analyzer + Clojure.
- 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring
  params desugar; unnamed, so self-recursion still resolves via the var.
- 00-syntax: completing that exposed a real gap — the overlay destructure
  fn didn't handle kwargs (a map pattern bound against a fn's sequential
  rest); it had only worked via the interpreter's destructure-bind. Added
  the seq->map coercion to the map? branch (sequential: 1 map elt => that
  map, else apply hash-map), matching destructure-bind so interpret ==
  compile.

Net: fn*/let*/loop* are plain-symbol primitives across interpreter,
analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro
params, incl kwargs & {:keys}) compiles through the macros with no
interpreter fallback. Regression spec added.

Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44,
clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all
unit+spec (destructuring 50/50).

* docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79)

grammar.ebnf: the destructuring note already attributed patterns to the
binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES
they desugar to take plain symbols only (a non-symbol binding errors, as
in Clojure).

self-hosting-compiler.md: the 'compile destructuring via a shared
destructure expander instead of falling back' item is done (and its
jolt-7dl ref was stale) — destructuring now compiles through the
fn/let/loop/defn macros' desugaring; the primitives reject patterns.

* core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy

First slice of moving stateful forms onto the compile path (jolt-eaa).
- loader stateful-head?: drop syntax-quote (the analyzer's `handled` set
  already compiles it; routing it to the interpreter was redundant).
- host_iface special-names: drop definterface/extend/proxy (stub macros
  expanding to def/nil — their expansions compile once unpunted).
- letfn stays interpreted: its let* expansion needs letrec semantics
  (mutual recursion between the fns), which sequential compiled let* lacks
  — a later tier.

Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + protocol/multimethod/macro specs.

---------

Co-authored-by: Yogthos <yogthos@gmail.com>

doc/grammar.ebnf

@@ -164,8 +164,11 @@ feature         = ":clj" | ":cljs" | ":default" | keyword ;
 tagged-literal  = "#" , symbol , ws , form ;
 
 (* ── Destructuring (semantics, not reader syntax) ────────────────────────────
-   The reader produces plain vectors and maps; the binding forms (let, fn, loop,
+   The reader produces plain vectors and maps; the binding MACROS (let, fn, loop,
-   doseq, for, defmacro params, …) interpret them as destructuring patterns:
+   doseq, for, defmacro params, …) interpret them as destructuring patterns. The
+   PRIMITIVES they desugar to — fn*, let*, loop* — take plain symbols ONLY (a
+   non-symbol binding is an error, as in Clojure: "fn params must be Symbols" /
+   "Bad binding form, expected symbol"). The grammar of a pattern:
 
      binding        = symbol | seq-binding | map-binding ;
      seq-binding    = "[" , { binding } , [ "&" , binding ] , [ ":as" , symbol ] , "]" ;

doc/self-hosting-compiler.md

@@ -150,9 +150,11 @@ linking is opt-in, never the default, so the REPL is always live.
 2. **Hybrid fallback + coverage** (`jolt-1bj`) — *done*. Forms the compiler can't
    compile throw `jolt/uncompilable` and fall back to the interpreter, so compile
    mode is always correct. Covered: multi-arity/named/variadic fns, `recur` in
-   `fn`, map/vector literals, and resolution matching the interpreter. (One
+   `fn`, map/vector literals, and resolution matching the interpreter.
-   optimization left: compile destructuring via a shared `destructure` expander
+   Destructuring compiles via the shared `destructure` expander: the `fn`/`let`/
-   instead of falling back — `jolt-7dl`.)
+   `loop`/`defn` macros desugar to plain-symbol `fn*`/`let*`/`loop*`, so it no
+   longer falls back — and the primitives reject patterns outright, matching
+   Clojure (`jolt-f79`).
 5. **Compile-by-default + AOT** (`jolt-7j9`) — *done, done out of order*. Once the
    hybrid path was validated at parity, compilation was flipped on by default and
    AOT images (`aot.janet`) landed. Done before 3–4 because it's the runtime

jolt-core/clojure/core/00-syntax.clj

@@ -102,11 +102,24 @@
                    (vloop 0 0 (conj (conj acc g) init)))
                (map? pat)
                  (let* [g (symbol (str (gensym)))
+                        gm (symbol (str (gensym)))
+                        ;; kwargs: a map pattern may bind against the sequential rest
+                        ;; of a fn — (& {:keys [...]}) — which is a seq of alternating
+                        ;; k/v args, or a single trailing map. Coerce like Clojure (and
+                        ;; like the interpreter's destructure-bind, so interpret/compile
+                        ;; agree): a sequential value with one map element is that map,
+                        ;; otherwise (apply hash-map). A real map value is used as-is, so
+                        ;; ordinary map destructuring is unaffected. g holds init once;
+                        ;; gm is the coerced map every lookup (and :as) reads from.
+                        coerce `(if (sequential? ~g)
+                                  (if (and (= 1 (count ~g)) (map? (first ~g)))
+                                    (first ~g)
+                                    (apply hash-map ~g))
+                                  ~g)
                         or-map (get pat :or)
                         as-sym (get pat :as)
-                        base (if as-sym
+                        bound (conj (conj (conj (conj acc g) init) gm) coerce)
-                               (conj (conj (conj (conj acc g) init) as-sym) g)
+                        base (if as-sym (conj (conj bound as-sym) gm) bound)
-                               (conj (conj acc g) init))
                         group
                           (fn* [a kw kind]
                             (let* [names (get pat kw)]
@@ -125,8 +138,8 @@
                                            fo (find-or or-map local)]
                                       (conj (conj aa (symbol local))
                                             (if (nth fo 0)
-                                              `(get ~g ~keyform ~(nth fo 1))
+                                              `(get ~gm ~keyform ~(nth fo 1))
-                                              `(get ~g ~keyform)))))
+                                              `(get ~gm ~keyform)))))
                                   a names)
                                 a)))
                         g1 (group base :keys :kw)
@@ -135,7 +148,7 @@
                    (reduce (fn* [a k]
                              (if (keyword? k)
                                a
-                               (proc k `(get ~g ~(get pat k)) a)))
+                               (proc k `(get ~gm ~(get pat k)) a)))
                            g3 (keys pat)))
                :else (throw (str "unsupported destructuring pattern"))))
          ploop
@@ -212,8 +225,11 @@
       (let [as (vec (map mk aftn))]
         (if nm `(fn* ~nm ~@as) `(fn* ~@as))))))
 
-;; defn: drop an optional leading docstring and attr-map, then (def name (fn* ...)).
+;; defn: drop an optional leading docstring and attr-map, then (def name (fn ...)).
-;; Both single- and multi-arity reduce to (fn* ~@body) — fn* takes either a params
+;; Emits the fn MACRO (not the fn* primitive) so destructuring params desugar — fn*
+;; requires plain symbols (like Clojure). Unnamed (as before): self-recursion
+;; resolves through the def'd var, so this only adds the desugaring step.
+;; Both single- and multi-arity reduce to (fn ~@body) — fn takes either a params
 ;; vector + body or a sequence of ([params] body) clauses, so no arity branching is
 ;; needed. (map? is true for symbol forms too, so guard the attr-map with symbol?.)
 ;; Defined before fresh-sym below, which is a defn-.
@@ -221,7 +237,7 @@
   (let [body (if (and (seq body) (string? (first body))) (rest body) body)
         body (if (and (seq body) (map? (first body)) (not (symbol? (first body))))
                (rest body) body)]
-    `(def ~fn-name (fn* ~@body))))
+    `(def ~fn-name (fn ~@body))))
 
 ;; Jolt doesn't enforce privacy, so defn- is just defn (matching how Clojure's own
 ;; defn- delegates to defn with :private metadata).

src/jolt/api.janet

@@ -14,14 +14,21 @@
 (import ./stdlib_embed :as stdlib-embed)
 (import ./host_iface :as host)
 
-# A defmacro expander compiles to a native fn (built as (fn* args body...) and run
+# A defmacro expander compiles to a native fn (built as (fn args body...) and run
-# through the self-hosted pipeline) so macro expansion is compiled, zero runtime
+# through the self-hosted pipeline) so macro expansion is COMPILED code, zero runtime
-# cost — instead of an interpreted closure. Returns nil (interpreted fallback) when
+# cost — instead of an interpreted closure, mirroring Clojure (macros are ordinary
-# the analyzer isn't built yet or the body isn't compilable.
+# compiled fns). Wrapped in the `fn` MACRO (not the `fn*` primitive) so a destructured
+# macro arglist — `[a & [b]]`, `[& {:keys [x]}]`, nested — desugars before lowering;
+# raw fn* punts on a destructuring rest param. Returns nil when the analyzer isn't
+# built yet (the early macros, expanded WHILE the analyzer is being bootstrapped) or
+# the body isn't compilable; in that case defmacro keeps an interpreted closure, and
+# backend/recompile-macros! replaces it with a compiled expander once the analyzer
+# comes alive (staged bootstrap — the interpreter is a build-time crutch, gone by
+# steady state).
 (set macro-compile-hook
   (fn [ctx args-form body]
     (backend/try-compile-fn ctx
-      (array/concat @[{:jolt/type :symbol :ns nil :name "fn*"} args-form] body))))
+      (array/concat @[{:jolt/type :symbol :ns nil :name "fn"} args-form] body))))
 
 (defn normalize-pvecs
   "Deep-convert any sequential (pvec/tuple/array) to a Janet tuple. Test helper
@@ -92,7 +99,12 @@
       # half-loaded core (which would forward-ref the missing kernel fns to nil).
       (when (tier :kernel) (put env :kernel-ready? true))))
   (put env :direct-linking? user-dl)
-  (ctx-set-current-ns ctx saved))
+  (ctx-set-current-ns ctx saved)
+  # Staged bootstrap: the early macros (00-syntax) were defined while the analyzer
+  # was still being built, so their expanders are interpreted closures. Now that the
+  # full overlay + analyzer are in place, recompile those expanders to native code —
+  # by steady state no macro expansion runs interpreted (no-op in interpreter mode).
+  (backend/ensure-macros-compiled! ctx))
 
 (defn init
   "Create a new Jolt evaluation context.

src/jolt/backend.janet

@@ -372,3 +372,49 @@
     (when (compiled 0)
       (def r (protect (eval (compiled 1) (comp/ctx-janet-env ctx))))
       (when (r 0) (r 1)))))
+
+# Wrap expanders in the `fn` MACRO, not the `fn*` primitive: `fn` desugars a
+# destructured macro arglist (`[a & [b]]`, `[& {:keys [x]}]`) before lowering,
+# whereas raw fn* punts on a destructuring rest param.
+(def- fn-sym {:jolt/type :symbol :ns nil :name "fn"})
+
+(defn recompile-macros!
+  "Staged-bootstrap second pass: once the self-hosted analyzer is alive, replace
+  every interpreted macro expander with a COMPILED one. The early macros (00-syntax
+  etc.) are defined WHILE the analyzer is still being bootstrapped, so their
+  expanders can't compile yet (the analyzer they'd compile through doesn't exist) —
+  defmacro gives them an interpreted closure as a build-time crutch and stashes the
+  source on the var (:macro-src). This pass compiles that source through the now-live
+  analyzer and rebinds the var, so by steady state no macro expansion is interpreted
+  — mirroring how a self-hosting compiler recompiles its seed once it can.
+
+  Idempotent: a var compiled once is marked :macro-compiled and skipped (so the
+  refer of a core macro into another ns, or a later rebuild, costs nothing). A macro
+  whose body uses &env/&form keeps its interpreted closure (the compiled fn* has no
+  such params). Returns the number of expanders compiled this pass."
+  [ctx]
+  (var n 0)
+  (each ns (all-ns ctx)
+    (each v (ns :mappings)
+      (when (and (var? v) (var-macro? v)
+                 (v :macro-src) (not (v :macro-compiled))
+                 (not (v :macro-uses-env)))
+        (def [args-form body] (v :macro-src))
+        (def compiled
+          (try-compile-fn ctx (array/concat @[fn-sym args-form] body)))
+        (when compiled
+          (bind-root v compiled)
+          (put v :macro-compiled true)
+          (++ n)))))
+  n)
+
+(defn ensure-macros-compiled!
+  "Called once the overlay is fully loaded (api/load-core-overlay!): in compile
+  mode, ensure the analyzer is built, then run the staged macro-recompile pass so
+  the early (interpreted-during-bootstrap) macro expanders become compiled. No-op
+  in interpreter mode (no analyzer, macros stay interpreted by design) and cheap to
+  call again (recompile-macros! skips already-compiled vars)."
+  [ctx]
+  (when (get (ctx :env) :compile?)
+    (ensure-analyzer ctx)
+    (when (analyzer-built? ctx) (recompile-macros! ctx))))

src/jolt/evaluator.janet

@@ -517,6 +517,21 @@
         (do (array/push fixed a) (+= i 1)))))
   {:fixed (tuple/slice (tuple ;fixed)) :rest rest-pat})
 
+(defn- plain-sym? [p] (and (struct? p) (= :symbol (p :jolt/type))))
+
+(defn- require-symbol-params
+  "fn* is a primitive: its params must be plain symbols. The fn/defn MACROS desugar
+  destructuring into plain params + a body let before emitting fn*, so fn* never
+  legitimately sees a pattern — matching Clojure, where (fn* [[a b]] ...) is the
+  compile error 'fn params must be Symbols'. Enforcing it here keeps the interpreter
+  consistent with the self-hosted analyzer (which also requires plain fn* params)
+  and with Clojure, instead of leniently destructuring a form Clojure rejects."
+  [param-info]
+  (each p (param-info :fixed)
+    (unless (plain-sym? p) (error "fn params must be Symbols")))
+  (let [r (param-info :rest)]
+    (when (and r (not (plain-sym? r))) (error "fn params must be Symbols"))))
+
 (defn- d-get
   "Look up key k in a map-like value (phm/struct/table/nil)."
   [m k]
@@ -805,6 +820,14 @@
                        ns (ctx-find-ns ctx ns-name)]
                    (def v (ns-intern ns (name-sym :name) macro-fn))
                    (put v :macro true)
+                   # Stash the expander source so backend/recompile-macros! can
+                   # compile it once the analyzer is alive (staged bootstrap): a
+                   # macro defined WHILE the analyzer is still being built gets an
+                   # interpreted closure now, a compiled expander later. uses-env
+                   # macros stay interpreted (the compiled fn* has no &env/&form).
+                   (put v :macro-src @[args-form body])
+                   (put v :macro-uses-env uses-env)
+                   (when compiled-fn (put v :macro-compiled true))
                    # A (re)defined macro invalidates any cached expansions.
                    (table/clear macro-cache)
                    (var-get v)))
@@ -917,6 +940,7 @@
                  (let [args-form (in pair 0)
                        body (tuple/slice pair 1)
                        param-info (parse-params args-form)
+                       _ (require-symbol-params param-info)
                        fixed-pats (param-info :fixed)
                        rest-pat (param-info :rest)
                        n-fixed (length fixed-pats)
@@ -954,6 +978,7 @@
              (let [args-form (in form 1)
                    body (tuple/slice form 2)
                    param-info (parse-params args-form)
+                   _ (require-symbol-params param-info)
                    fixed-pats (param-info :fixed)
                    rest-pat (param-info :rest)
                    defining-ns (ctx-current-ns ctx)]
@@ -986,6 +1011,9 @@
               (let [len (length bind-vec)]
                 (while (< i len)
                   (let [pat (bind-vec i)]
+                    # let* is a primitive (the let macro desugars destructuring);
+                    # its binding names must be plain symbols, as in Clojure.
+                    (unless (plain-sym? pat) (error "Bad binding form, expected symbol"))
                     (def val (eval-form ctx new-bindings (bind-vec (+ i 1))))
                     (destructure-bind ctx new-bindings pat val)
                     (+= i 2))))
@@ -999,8 +1027,10 @@
                   patterns @[]]
               (var i 0)
               (while (< i (length bind-vec))
+                # loop* is a primitive (the loop macro desugars destructuring);
+                # its binding names must be plain symbols, as in Clojure.
+                (unless (plain-sym? (bind-vec i)) (error "Bad binding form, expected symbol"))
                 (array/push init-vals (eval-form ctx bindings (bind-vec (+ i 1))))
-                # keep the binding form (symbol OR destructuring pattern)
                 (array/push patterns (bind-vec i))
                 (+= i 2))
               (var loop-fn nil)

src/jolt/host_iface.janet

@@ -81,8 +81,10 @@
              "create-ns" "remove-ns" "find-ns" "all-ns" "the-ns" "resolve"
              "ns-resolve" "ns-aliases" "ns-imports" "ns-interns"
              "read-string" "macroexpand-1" "defonce" "ns" "in-ns" "require"
-             "import" "use" "refer" "defrecord" "defprotocol" "definterface"
+             "import" "use" "refer" "defrecord" "defprotocol"
-             "reify" "proxy" "extend-type" "extend-protocol" "extend" "gen-class"
+             "reify" "extend-type" "extend-protocol" "gen-class"
+             # letfn stays: its let* expansion needs letrec semantics (mutual
+             # recursion between the fns), which compiled sequential let* lacks.
              "monitor-enter" "monitor-exit" "binding" "letfn"]
       (put t n true))
     t))

src/jolt/loader.janet

@@ -9,12 +9,14 @@
 # Stateful / context-modifying forms always interpret: they mutate the context
 # (namespaces, macros, types, multimethods, dynamic vars, …) in ways the compiler
 # doesn't model. Kept here so the compile/interpret routing lives in one place,
-# used by both load-ns and the public eval-one.
+# used by both load-ns and the public eval-one. Shrinking toward the frozen
+# host-coupled set (Stage 2 jolt-eaa): forms move off this list as they gain a
+# compile path; syntax-quote already compiles via the analyzer's `handled` set.
 (defn- stateful-head? [head-name]
   (or (= head-name "defmacro") (= head-name "ns")
       (= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod")
       (= head-name "require") (= head-name "in-ns")
-      (= head-name "syntax-quote") (= head-name "set!")
+      (= head-name "set!")
       (= head-name "var") (= head-name ".") (= head-name "new")
       (= head-name "eval")))
 

test/spec/destructuring-spec.janet

@@ -55,6 +55,20 @@
   ["loop map binding"   "4"      "(loop [{:keys [v]} {:v 1} n 0] (if (< n 2) (recur {:v (* v 2)} (inc n)) v))"]
   ["loop init sees destr" "[1 2 3]" "(loop [[a b] [1 2] c (+ a b)] [a b c])"])
 
+# fn*/let*/loop* are PRIMITIVES: their binding forms must be plain symbols, exactly
+# as in Clojure ("fn params must be Symbols" / "Bad binding form, expected symbol").
+# Destructuring lives in the fn/let/loop/defn MACROS, which desugar to plain
+# primitives — so the interpreter and the self-hosted analyzer agree (neither
+# destructures fn*), and nothing silently falls back to a lenient interpreter path.
+(defspec "destructure / primitives reject patterns (jolt-f79)"
+  ["fn* fixed pattern"  :throws  "((fn* [[a b]] a) [1 2])"]
+  ["fn* rest pattern"   :throws  "((fn* [a & [b]] b) 1 2 3)"]
+  ["let* pattern"       :throws  "(let* [[a b] [1 2]] a)"]
+  ["loop* pattern"      :throws  "(loop* [[a b] [1 2]] a)"]
+  # the macros still desugar the same patterns to plain primitives:
+  ["fn desugars"        "[1 2]"  "((fn [[a b]] [a b]) [1 2])"]
+  ["let desugars"       "[1 2]"  "(let [[a b] [1 2]] [a b])"])
+
 (defspec "destructure / macro params"
   ["macro & [a & more :as all]"
    "[1 [2 3] [1 2 3]]"