From 534007641ef1aa541a69024bd332a9fc67ae01aa Mon Sep 17 00:00:00 2001 From: Dmitri Sotnikov Date: Wed, 10 Jun 2026 02:20:38 +0800 Subject: [PATCH] Stage2 compile only (#12) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * core: compile macro expanders via staged bootstrap (Stage 2 Task 1) Macros are now compiled, not interpreted, by steady state — matching Clojure (macros are ordinary compiled fns the Java seed compiles for clojure.core) and ClojureScript (macros compiled, invoked at compile time). Neither reference keeps an interpreted-closure fallback. The early macros (00-syntax) are defined while the self-hosted analyzer is still being bootstrapped (it builds lazily after the overlay loads), so macro-compile-hook returns nil and they get an interpreted closure. The bootstrap compiler.janet can't substitute (it punts on syntax-quote, which nearly every expander uses). Fix = staged bootstrap, the same pattern as the compiler fixpoint: defmacro stashes the expander source on the var (:macro-src) plus a :macro-uses-env flag; once the overlay + analyzer are fully built, backend/recompile-macros! (via ensure-macros-compiled! at the end of load-core-overlay!) compiles each stashed expander through the now-live analyzer and rebinds the var, marking :macro-compiled. Idempotent; &env/&form macros keep the interpreted closure (the compiled fn* has no such params). The interpreter is now a build-time crutch, gone by steady state. Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring (which the analyzer punts on) to a plain rest param + (first rest), so all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79. 47/47 overlay macros compiled, 0 interpreted; user macros compile immediately post-init. Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all unit+spec, core-bench neutral. * core: wrap macro expanders in fn (not fn*) so destructured arglists compile Follow-up to the staged macro-compile change. The macro-recompile pass wrapped each expander in the raw fn* primitive, which punts on a destructuring rest param — so if-not/if-let/if-some/assert (using the canonical '& [else]') couldn't compile and had been rewritten to plain rest params as a workaround. Root cause: fn* is the primitive; the fn MACRO is what desugars destructuring (rest, map, nested) into the body before lowering. Wrapping expanders in fn instead of fn* compiles any destructured macro arglist uniformly, so the workaround is unnecessary — reverted those 4 macros to the canonical '& [else]' forms. Net: rest-destructuring is fully compiled for all normal code (fn/defn/ let/macro params). Only the hand-written raw fn* primitive still punts (jolt-f79, downgraded to P4 — falls back to interpreter, still correct). 47/47 overlay macros compiled. Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap, features, all unit+spec. * core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79) jolt-f79 asked to compile destructuring in fn* rest params. Checking against Clojure inverts the premise: Clojure's fn* REJECTS destructuring at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding form, expected symbol'). So the self-hosted analyzer was already correct — fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn MACROS desugar destructuring. The real defect was the interpreter leniently destructuring raw fn*, and defn emitting raw fn* to rely on it. Changes: - evaluator: fn*/let*/loop* now reject non-symbol binding forms with Clojure's exact messages (require-symbol-params/plain-sym?), so the interpreter agrees with the analyzer + Clojure. - 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring params desugar; unnamed, so self-recursion still resolves via the var. - 00-syntax: completing that exposed a real gap — the overlay destructure fn didn't handle kwargs (a map pattern bound against a fn's sequential rest); it had only worked via the interpreter's destructure-bind. Added the seq->map coercion to the map? branch (sequential: 1 map elt => that map, else apply hash-map), matching destructure-bind so interpret == compile. Net: fn*/let*/loop* are plain-symbol primitives across interpreter, analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro params, incl kwargs & {:keys}) compiles through the macros with no interpreter fallback. Regression spec added. Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all unit+spec (destructuring 50/50). * docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79) grammar.ebnf: the destructuring note already attributed patterns to the binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES they desugar to take plain symbols only (a non-symbol binding errors, as in Clojure). self-hosting-compiler.md: the 'compile destructuring via a shared destructure expander instead of falling back' item is done (and its jolt-7dl ref was stale) — destructuring now compiles through the fn/let/loop/defn macros' desugaring; the primitives reject patterns. * core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy First slice of moving stateful forms onto the compile path (jolt-eaa). - loader stateful-head?: drop syntax-quote (the analyzer's `handled` set already compiles it; routing it to the interpreter was redundant). - host_iface special-names: drop definterface/extend/proxy (stub macros expanding to def/nil — their expansions compile once unpunted). - letfn stays interpreted: its let* expansion needs letrec semantics (mutual recursion between the fns), which sequential compiled let* lacks — a later tier. Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67, features 78/78, all unit + protocol/multimethod/macro specs. --------- Co-authored-by: Yogthos --- doc/grammar.ebnf | 7 +++-- doc/self-hosting-compiler.md | 8 +++-- jolt-core/clojure/core/00-syntax.clj | 34 ++++++++++++++------ src/jolt/api.janet | 24 +++++++++++---- src/jolt/backend.janet | 46 ++++++++++++++++++++++++++++ src/jolt/evaluator.janet | 32 ++++++++++++++++++- src/jolt/host_iface.janet | 6 ++-- src/jolt/loader.janet | 6 ++-- test/spec/destructuring-spec.janet | 14 +++++++++ 9 files changed, 152 insertions(+), 25 deletions(-) diff --git a/doc/grammar.ebnf b/doc/grammar.ebnf index 4c2f30e..dffffb1 100644 --- a/doc/grammar.ebnf +++ b/doc/grammar.ebnf @@ -164,8 +164,11 @@ feature = ":clj" | ":cljs" | ":default" | keyword ; tagged-literal = "#" , symbol , ws , form ; (* ── Destructuring (semantics, not reader syntax) ──────────────────────────── - The reader produces plain vectors and maps; the binding forms (let, fn, loop, - doseq, for, defmacro params, …) interpret them as destructuring patterns: + The reader produces plain vectors and maps; the binding MACROS (let, fn, loop, + doseq, for, defmacro params, …) interpret them as destructuring patterns. The + PRIMITIVES they desugar to — fn*, let*, loop* — take plain symbols ONLY (a + non-symbol binding is an error, as in Clojure: "fn params must be Symbols" / + "Bad binding form, expected symbol"). The grammar of a pattern: binding = symbol | seq-binding | map-binding ; seq-binding = "[" , { binding } , [ "&" , binding ] , [ ":as" , symbol ] , "]" ; diff --git a/doc/self-hosting-compiler.md b/doc/self-hosting-compiler.md index 05d9ada..ea21c56 100644 --- a/doc/self-hosting-compiler.md +++ b/doc/self-hosting-compiler.md @@ -150,9 +150,11 @@ linking is opt-in, never the default, so the REPL is always live. 2. **Hybrid fallback + coverage** (`jolt-1bj`) — *done*. Forms the compiler can't compile throw `jolt/uncompilable` and fall back to the interpreter, so compile mode is always correct. Covered: multi-arity/named/variadic fns, `recur` in - `fn`, map/vector literals, and resolution matching the interpreter. (One - optimization left: compile destructuring via a shared `destructure` expander - instead of falling back — `jolt-7dl`.) + `fn`, map/vector literals, and resolution matching the interpreter. + Destructuring compiles via the shared `destructure` expander: the `fn`/`let`/ + `loop`/`defn` macros desugar to plain-symbol `fn*`/`let*`/`loop*`, so it no + longer falls back — and the primitives reject patterns outright, matching + Clojure (`jolt-f79`). 5. **Compile-by-default + AOT** (`jolt-7j9`) — *done, done out of order*. Once the hybrid path was validated at parity, compilation was flipped on by default and AOT images (`aot.janet`) landed. Done before 3–4 because it's the runtime diff --git a/jolt-core/clojure/core/00-syntax.clj b/jolt-core/clojure/core/00-syntax.clj index aeb33f2..1b3131c 100644 --- a/jolt-core/clojure/core/00-syntax.clj +++ b/jolt-core/clojure/core/00-syntax.clj @@ -102,11 +102,24 @@ (vloop 0 0 (conj (conj acc g) init))) (map? pat) (let* [g (symbol (str (gensym))) + gm (symbol (str (gensym))) + ;; kwargs: a map pattern may bind against the sequential rest + ;; of a fn — (& {:keys [...]}) — which is a seq of alternating + ;; k/v args, or a single trailing map. Coerce like Clojure (and + ;; like the interpreter's destructure-bind, so interpret/compile + ;; agree): a sequential value with one map element is that map, + ;; otherwise (apply hash-map). A real map value is used as-is, so + ;; ordinary map destructuring is unaffected. g holds init once; + ;; gm is the coerced map every lookup (and :as) reads from. + coerce `(if (sequential? ~g) + (if (and (= 1 (count ~g)) (map? (first ~g))) + (first ~g) + (apply hash-map ~g)) + ~g) or-map (get pat :or) as-sym (get pat :as) - base (if as-sym - (conj (conj (conj (conj acc g) init) as-sym) g) - (conj (conj acc g) init)) + bound (conj (conj (conj (conj acc g) init) gm) coerce) + base (if as-sym (conj (conj bound as-sym) gm) bound) group (fn* [a kw kind] (let* [names (get pat kw)] @@ -125,8 +138,8 @@ fo (find-or or-map local)] (conj (conj aa (symbol local)) (if (nth fo 0) - `(get ~g ~keyform ~(nth fo 1)) - `(get ~g ~keyform))))) + `(get ~gm ~keyform ~(nth fo 1)) + `(get ~gm ~keyform))))) a names) a))) g1 (group base :keys :kw) @@ -135,7 +148,7 @@ (reduce (fn* [a k] (if (keyword? k) a - (proc k `(get ~g ~(get pat k)) a))) + (proc k `(get ~gm ~(get pat k)) a))) g3 (keys pat))) :else (throw (str "unsupported destructuring pattern")))) ploop @@ -212,8 +225,11 @@ (let [as (vec (map mk aftn))] (if nm `(fn* ~nm ~@as) `(fn* ~@as)))))) -;; defn: drop an optional leading docstring and attr-map, then (def name (fn* ...)). -;; Both single- and multi-arity reduce to (fn* ~@body) — fn* takes either a params +;; defn: drop an optional leading docstring and attr-map, then (def name (fn ...)). +;; Emits the fn MACRO (not the fn* primitive) so destructuring params desugar — fn* +;; requires plain symbols (like Clojure). Unnamed (as before): self-recursion +;; resolves through the def'd var, so this only adds the desugaring step. +;; Both single- and multi-arity reduce to (fn ~@body) — fn takes either a params ;; vector + body or a sequence of ([params] body) clauses, so no arity branching is ;; needed. (map? is true for symbol forms too, so guard the attr-map with symbol?.) ;; Defined before fresh-sym below, which is a defn-. @@ -221,7 +237,7 @@ (let [body (if (and (seq body) (string? (first body))) (rest body) body) body (if (and (seq body) (map? (first body)) (not (symbol? (first body)))) (rest body) body)] - `(def ~fn-name (fn* ~@body)))) + `(def ~fn-name (fn ~@body)))) ;; Jolt doesn't enforce privacy, so defn- is just defn (matching how Clojure's own ;; defn- delegates to defn with :private metadata). diff --git a/src/jolt/api.janet b/src/jolt/api.janet index b97fa28..fe10f27 100644 --- a/src/jolt/api.janet +++ b/src/jolt/api.janet @@ -14,14 +14,21 @@ (import ./stdlib_embed :as stdlib-embed) (import ./host_iface :as host) -# A defmacro expander compiles to a native fn (built as (fn* args body...) and run -# through the self-hosted pipeline) so macro expansion is compiled, zero runtime -# cost — instead of an interpreted closure. Returns nil (interpreted fallback) when -# the analyzer isn't built yet or the body isn't compilable. +# A defmacro expander compiles to a native fn (built as (fn args body...) and run +# through the self-hosted pipeline) so macro expansion is COMPILED code, zero runtime +# cost — instead of an interpreted closure, mirroring Clojure (macros are ordinary +# compiled fns). Wrapped in the `fn` MACRO (not the `fn*` primitive) so a destructured +# macro arglist — `[a & [b]]`, `[& {:keys [x]}]`, nested — desugars before lowering; +# raw fn* punts on a destructuring rest param. Returns nil when the analyzer isn't +# built yet (the early macros, expanded WHILE the analyzer is being bootstrapped) or +# the body isn't compilable; in that case defmacro keeps an interpreted closure, and +# backend/recompile-macros! replaces it with a compiled expander once the analyzer +# comes alive (staged bootstrap — the interpreter is a build-time crutch, gone by +# steady state). (set macro-compile-hook (fn [ctx args-form body] (backend/try-compile-fn ctx - (array/concat @[{:jolt/type :symbol :ns nil :name "fn*"} args-form] body)))) + (array/concat @[{:jolt/type :symbol :ns nil :name "fn"} args-form] body)))) (defn normalize-pvecs "Deep-convert any sequential (pvec/tuple/array) to a Janet tuple. Test helper @@ -92,7 +99,12 @@ # half-loaded core (which would forward-ref the missing kernel fns to nil). (when (tier :kernel) (put env :kernel-ready? true)))) (put env :direct-linking? user-dl) - (ctx-set-current-ns ctx saved)) + (ctx-set-current-ns ctx saved) + # Staged bootstrap: the early macros (00-syntax) were defined while the analyzer + # was still being built, so their expanders are interpreted closures. Now that the + # full overlay + analyzer are in place, recompile those expanders to native code — + # by steady state no macro expansion runs interpreted (no-op in interpreter mode). + (backend/ensure-macros-compiled! ctx)) (defn init "Create a new Jolt evaluation context. diff --git a/src/jolt/backend.janet b/src/jolt/backend.janet index fe797cb..9352517 100644 --- a/src/jolt/backend.janet +++ b/src/jolt/backend.janet @@ -372,3 +372,49 @@ (when (compiled 0) (def r (protect (eval (compiled 1) (comp/ctx-janet-env ctx)))) (when (r 0) (r 1))))) + +# Wrap expanders in the `fn` MACRO, not the `fn*` primitive: `fn` desugars a +# destructured macro arglist (`[a & [b]]`, `[& {:keys [x]}]`) before lowering, +# whereas raw fn* punts on a destructuring rest param. +(def- fn-sym {:jolt/type :symbol :ns nil :name "fn"}) + +(defn recompile-macros! + "Staged-bootstrap second pass: once the self-hosted analyzer is alive, replace + every interpreted macro expander with a COMPILED one. The early macros (00-syntax + etc.) are defined WHILE the analyzer is still being bootstrapped, so their + expanders can't compile yet (the analyzer they'd compile through doesn't exist) — + defmacro gives them an interpreted closure as a build-time crutch and stashes the + source on the var (:macro-src). This pass compiles that source through the now-live + analyzer and rebinds the var, so by steady state no macro expansion is interpreted + — mirroring how a self-hosting compiler recompiles its seed once it can. + + Idempotent: a var compiled once is marked :macro-compiled and skipped (so the + refer of a core macro into another ns, or a later rebuild, costs nothing). A macro + whose body uses &env/&form keeps its interpreted closure (the compiled fn* has no + such params). Returns the number of expanders compiled this pass." + [ctx] + (var n 0) + (each ns (all-ns ctx) + (each v (ns :mappings) + (when (and (var? v) (var-macro? v) + (v :macro-src) (not (v :macro-compiled)) + (not (v :macro-uses-env))) + (def [args-form body] (v :macro-src)) + (def compiled + (try-compile-fn ctx (array/concat @[fn-sym args-form] body))) + (when compiled + (bind-root v compiled) + (put v :macro-compiled true) + (++ n))))) + n) + +(defn ensure-macros-compiled! + "Called once the overlay is fully loaded (api/load-core-overlay!): in compile + mode, ensure the analyzer is built, then run the staged macro-recompile pass so + the early (interpreted-during-bootstrap) macro expanders become compiled. No-op + in interpreter mode (no analyzer, macros stay interpreted by design) and cheap to + call again (recompile-macros! skips already-compiled vars)." + [ctx] + (when (get (ctx :env) :compile?) + (ensure-analyzer ctx) + (when (analyzer-built? ctx) (recompile-macros! ctx)))) diff --git a/src/jolt/evaluator.janet b/src/jolt/evaluator.janet index 32ce62c..dfcae00 100644 --- a/src/jolt/evaluator.janet +++ b/src/jolt/evaluator.janet @@ -517,6 +517,21 @@ (do (array/push fixed a) (+= i 1))))) {:fixed (tuple/slice (tuple ;fixed)) :rest rest-pat}) +(defn- plain-sym? [p] (and (struct? p) (= :symbol (p :jolt/type)))) + +(defn- require-symbol-params + "fn* is a primitive: its params must be plain symbols. The fn/defn MACROS desugar + destructuring into plain params + a body let before emitting fn*, so fn* never + legitimately sees a pattern — matching Clojure, where (fn* [[a b]] ...) is the + compile error 'fn params must be Symbols'. Enforcing it here keeps the interpreter + consistent with the self-hosted analyzer (which also requires plain fn* params) + and with Clojure, instead of leniently destructuring a form Clojure rejects." + [param-info] + (each p (param-info :fixed) + (unless (plain-sym? p) (error "fn params must be Symbols"))) + (let [r (param-info :rest)] + (when (and r (not (plain-sym? r))) (error "fn params must be Symbols")))) + (defn- d-get "Look up key k in a map-like value (phm/struct/table/nil)." [m k] @@ -805,6 +820,14 @@ ns (ctx-find-ns ctx ns-name)] (def v (ns-intern ns (name-sym :name) macro-fn)) (put v :macro true) + # Stash the expander source so backend/recompile-macros! can + # compile it once the analyzer is alive (staged bootstrap): a + # macro defined WHILE the analyzer is still being built gets an + # interpreted closure now, a compiled expander later. uses-env + # macros stay interpreted (the compiled fn* has no &env/&form). + (put v :macro-src @[args-form body]) + (put v :macro-uses-env uses-env) + (when compiled-fn (put v :macro-compiled true)) # A (re)defined macro invalidates any cached expansions. (table/clear macro-cache) (var-get v))) @@ -917,6 +940,7 @@ (let [args-form (in pair 0) body (tuple/slice pair 1) param-info (parse-params args-form) + _ (require-symbol-params param-info) fixed-pats (param-info :fixed) rest-pat (param-info :rest) n-fixed (length fixed-pats) @@ -954,6 +978,7 @@ (let [args-form (in form 1) body (tuple/slice form 2) param-info (parse-params args-form) + _ (require-symbol-params param-info) fixed-pats (param-info :fixed) rest-pat (param-info :rest) defining-ns (ctx-current-ns ctx)] @@ -986,6 +1011,9 @@ (let [len (length bind-vec)] (while (< i len) (let [pat (bind-vec i)] + # let* is a primitive (the let macro desugars destructuring); + # its binding names must be plain symbols, as in Clojure. + (unless (plain-sym? pat) (error "Bad binding form, expected symbol")) (def val (eval-form ctx new-bindings (bind-vec (+ i 1)))) (destructure-bind ctx new-bindings pat val) (+= i 2)))) @@ -999,8 +1027,10 @@ patterns @[]] (var i 0) (while (< i (length bind-vec)) + # loop* is a primitive (the loop macro desugars destructuring); + # its binding names must be plain symbols, as in Clojure. + (unless (plain-sym? (bind-vec i)) (error "Bad binding form, expected symbol")) (array/push init-vals (eval-form ctx bindings (bind-vec (+ i 1)))) - # keep the binding form (symbol OR destructuring pattern) (array/push patterns (bind-vec i)) (+= i 2)) (var loop-fn nil) diff --git a/src/jolt/host_iface.janet b/src/jolt/host_iface.janet index d2dbd9e..96574c7 100644 --- a/src/jolt/host_iface.janet +++ b/src/jolt/host_iface.janet @@ -81,8 +81,10 @@ "create-ns" "remove-ns" "find-ns" "all-ns" "the-ns" "resolve" "ns-resolve" "ns-aliases" "ns-imports" "ns-interns" "read-string" "macroexpand-1" "defonce" "ns" "in-ns" "require" - "import" "use" "refer" "defrecord" "defprotocol" "definterface" - "reify" "proxy" "extend-type" "extend-protocol" "extend" "gen-class" + "import" "use" "refer" "defrecord" "defprotocol" + "reify" "extend-type" "extend-protocol" "gen-class" + # letfn stays: its let* expansion needs letrec semantics (mutual + # recursion between the fns), which compiled sequential let* lacks. "monitor-enter" "monitor-exit" "binding" "letfn"] (put t n true)) t)) diff --git a/src/jolt/loader.janet b/src/jolt/loader.janet index a2b9d07..f7608ea 100644 --- a/src/jolt/loader.janet +++ b/src/jolt/loader.janet @@ -9,12 +9,14 @@ # Stateful / context-modifying forms always interpret: they mutate the context # (namespaces, macros, types, multimethods, dynamic vars, …) in ways the compiler # doesn't model. Kept here so the compile/interpret routing lives in one place, -# used by both load-ns and the public eval-one. +# used by both load-ns and the public eval-one. Shrinking toward the frozen +# host-coupled set (Stage 2 jolt-eaa): forms move off this list as they gain a +# compile path; syntax-quote already compiles via the analyzer's `handled` set. (defn- stateful-head? [head-name] (or (= head-name "defmacro") (= head-name "ns") (= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod") (= head-name "require") (= head-name "in-ns") - (= head-name "syntax-quote") (= head-name "set!") + (= head-name "set!") (= head-name "var") (= head-name ".") (= head-name "new") (= head-name "eval"))) diff --git a/test/spec/destructuring-spec.janet b/test/spec/destructuring-spec.janet index 31d8afd..91525dd 100644 --- a/test/spec/destructuring-spec.janet +++ b/test/spec/destructuring-spec.janet @@ -55,6 +55,20 @@ ["loop map binding" "4" "(loop [{:keys [v]} {:v 1} n 0] (if (< n 2) (recur {:v (* v 2)} (inc n)) v))"] ["loop init sees destr" "[1 2 3]" "(loop [[a b] [1 2] c (+ a b)] [a b c])"]) +# fn*/let*/loop* are PRIMITIVES: their binding forms must be plain symbols, exactly +# as in Clojure ("fn params must be Symbols" / "Bad binding form, expected symbol"). +# Destructuring lives in the fn/let/loop/defn MACROS, which desugar to plain +# primitives — so the interpreter and the self-hosted analyzer agree (neither +# destructures fn*), and nothing silently falls back to a lenient interpreter path. +(defspec "destructure / primitives reject patterns (jolt-f79)" + ["fn* fixed pattern" :throws "((fn* [[a b]] a) [1 2])"] + ["fn* rest pattern" :throws "((fn* [a & [b]] b) 1 2 3)"] + ["let* pattern" :throws "(let* [[a b] [1 2]] a)"] + ["loop* pattern" :throws "(loop* [[a b] [1 2]] a)"] + # the macros still desugar the same patterns to plain primitives: + ["fn desugars" "[1 2]" "((fn [[a b]] [a b]) [1 2])"] + ["let desugars" "[1 2]" "(let [[a b] [1 2]] [a b])"]) + (defspec "destructure / macro params" ["macro & [a & more :as all]" "[1 [2 3] [1 2 3]]"