* compiler: self-hosted analyzer compiles set literals (#{…})
Stage 1 Task 1. analyzer.clj punted set literals to the interpreter
((form-set? form) (uncompilable "set literal")); now it builds the set-node IR
(already defined in ir.clj) from (form-set-items form), and backend.janet emits
(make-phs e1 e2 …) — each element evaluated then the persistent set built,
mirroring compiler.janet's emit-set-expr and the interpreter's :jolt/set path.
Closes a self-hosted-analyzer vs bootstrap-compiler parity gap: #{…} no longer
forces interpreter fallback on the compile path.
Gate: conformance 262x3 (+4 set-literal cases incl computed elements / empty /
in-let), fixpoint, self-host, sci, suite 3981/66, specs+unit green; core-bench
neutral (A/B). set?/disj-as-fns remain deliberately interpreted (in-sync across
all three lists) — adjudicated in Task 2.
* test: fallback-zero harness — assert non-stateful forms compile (not interpret)
Stage 1 Task 3. self-host-test checks results but not which path ran. This runs
the portable analyzer (backend/analyze-form) on a corpus of non-stateful forms
and asserts NONE raise :jolt/uncompilable — i.e. the self-hosted analyzer
compiled them, not the interpreter fallback. Inverse sanity list confirms a few
intentional-interpret forms (ns/defmacro/require/set?/letfn) still punt, so the
harness can't pass by compiling everything.
29 must-compile (incl set literals from Task 1) + 5 must-punt, 0 failures. As
Stage 1 parity grows, forms move from the punt list into must-compile; when the
fallback set equals the frozen intentional stateful set, the bootstrap is
retireable.
* core: migrate 7 lazy seq fns from the Janet seed to the Clojure overlay (40-lazy)
Finishes a port the prior team started and reverted (bb4a3e0): the 40-lazy.clj
tier moved lazy seq fns Janet→Clojure but regressed the suite to 849 because
lazy-seq's expansion leaked as data in compile mode — that was jolt-r81, since
root-fixed (lazy-seq/lazy-cat moved to 00-syntax). With the wall gone, the port
works. This shrinks the Janet seed toward the north star (self-hosted
clojure-in-clojure on a minimal host bootstrap).
Moved to core/40-lazy.clj (wired as a loaded tier after 30-macros):
distinct keep keep-indexed map-indexed cycle repeat iterate
40-lazy.clj completed to full parity: distinct gains its transducer arity;
keep/keep-indexed/map-indexed already had both arities.
Removed from the Janet seed (core.janet): the 7 core-* fns + their core-bindings
entries, the now-dead td-keep/td-map-indexed transducer helpers (the CLJ versions
carry their own), and the already-dead core-partition-by/core-xml-seq (shadowed by
10-seq/20-coll). Net: core.janet −131 lines.
Deferred (kept in Janet, separate follow-ups): partition-all (a CLJ port via
take/drop realizes a non-minimal element count, tripping the §6.3 laziness
counters + a suite file) and repeatedly (canonical CLJ doesn't validate args, so
the repeatedly.cljc throw cases regress). Both need behavior-matching first.
Gate: conformance 262x3, lazy-infinite 44/44, clojure-test-suite 4004/66 (UP from
3981 — the CLJ versions add coverage, e.g. distinct value-equality), fixpoint,
self-host, sci 422/0, fallback-zero, specs+unit, core-bench all green.
* test: raise clojure-test-suite baseline 3981 -> 4004 (lazy-fn migration coverage)
* core: migrate partition-all to the Clojure overlay (minimal realization)
Resolves the deferred partition-all port (jolt-yo3). The earlier CLJ attempt via
lazy take/drop over-realized vs the Janet pstep, tripping the §6.3 laziness
counter. The collection arities now realize EXACTLY n per chunk with a first/rest
loop and continue from the advanced cursor (no re-drop), so (take 3 (partition-all
2 (map counting (range)))) realizes exactly 6 — matching minimal realization in
both interpret and compile modes. Keeps transducer + [n coll] + [n step coll]
arities. letfn-bound recursion sidesteps the compile-mode multi-arity closure bug
(jolt-zxw), like keep-indexed/map-indexed.
Removed from the Janet seed: core-partition-all + its binding + the now-dead
td-partition-all helper (the CLJ version carries its own transducer arity).
Gate: conformance 262x3, lazy-infinite 44/44 (incl the §6.3 partition-all
counter), clojure-test-suite 4004/66, fixpoint, self-host, specs+unit green.
* core: migrate repeatedly to Clojure + fix char-not-callable / take count validation
Resolves the deferred repeatedly port (jolt-8qx). The blockers were two jolt
leniencies vs Clojure, now fixed (and correct beyond repeatedly):
- A char (a :jolt/type-tagged struct) fell into the struct-as-map branch of both
jolt-call (compile path) and the interpreter's apply dispatch, so (\a) returned
nil instead of throwing. Now only an UNtagged struct (a map literal) — or a
record — is callable as a key lookup; tagged structs fall through to "Cannot
call … as a function". Symbols are still handled (keyword-style get).
- core-take didn't validate its count, letting Janet's >= silently compare an int
to a char/string. It now rejects a non-number n like Clojure.
With those, the canonical CLJ repeatedly matches: (first (repeatedly non-fn)) and
(repeatedly non-number f) throw. Moved repeatedly to core/40-lazy.clj; removed
core-repeatedly + its binding from the seed.
These correctness fixes help broadly: repeatedly.cljc goes clean (19/10 -> 29/0),
and the suite rises 4004 -> 4034 pass / 66 -> 67 clean. Baseline raised.
Gate: conformance 262x3, lazy-infinite 44/44, clojure-test-suite 4034/67,
fixpoint, self-host, sci, fallback-zero, specs+unit green.
* test: document the 4004 -> 4034 baseline raise (partition-all/repeatedly + char/take fixes)
* docs: partition-all letfn is for minimal realization, not jolt-zxw
jolt-zxw (multi-arity arity-param mis-capture in a nested lazy-seq under :compile?)
is no longer reproducible: an arity-direct partition-all now compiles correctly in
the overlay. The original failure was an artifact of the CLJ multi-arity version
coexisting with the Janet core-partition-all ([n & rest]) during migration — the
shadowing confused multi-arity dispatch; removing the Janet version resolved it.
The letfn in partition-all stays purely for minimal realization (jolt-yo3).
* compiler: compile set?/disj as plain fns (close the last Stage-1 fallback gap)
Stage 1 jolt-g3h. set? and disj were special-cased in all three "can't compile"
lists (host_iface special-names, compiler.janet uncompilable-heads, evaluator
special-symbol? + handlers) — but they're pure value-production with callable
core vars (core-set?/core-disj), and those vars are byte-for-byte equivalent to
the evaluator handlers. Removed them from all three lists + dropped the now-dead
evaluator handler arms, so they're ordinary clojure.core fns everywhere: the
analyzer compiles (set? x)/(disj s x) as normal var calls instead of punting to
the interpreter.
Verified identical results in default AND JOLT_MUTABLE builds (no representation
sensitivity — sets are phs in both, unlike vector?/list? which collapse).
With this, the self-hosted analyzer's compile-path fallback set equals the frozen
intentional stateful set (Task 2) — it's now a strict superset of the bootstrap
compiler's compilable surface, so the Janet bootstrap is retireable (Stage 2).
fallback-zero: set?/disj moved to must-compile (31 now), set! into must-punt.
Gate: conformance 267x3 (+5 set?/disj cases), lazy-infinite 44/44, suite 4034/67,
fixpoint, self-host, sci, specs+unit green.
212 lines
9.3 KiB
Clojure
212 lines
9.3 KiB
Clojure
(ns jolt.analyzer
|
|
"Portable Clojure analyzer: reader form -> host-neutral IR (see jolt.ir).
|
|
|
|
Pure jolt-core — depends only on the host contract (jolt.host) and IR
|
|
constructors (jolt.ir), never on Janet. The contract fns are referred unqualified
|
|
(host form predicates are `form-*` to avoid colliding with clojure.core), so the
|
|
bootstrap can compile this namespace via its plain :var path. ctx is an opaque
|
|
host handle threaded to the contract fns; the analyzer never inspects it.
|
|
|
|
Coverage grows toward compiler.janet; unsupported forms throw :jolt/uncompilable
|
|
so the caller falls back to the interpreter (the hybrid contract).
|
|
|
|
`env` carries lexical state: {:locals #{names} :recur recur-target-name|nil}.
|
|
Definitions are ordered so only `analyze` (mutually recursive) is forward
|
|
declared — the bootstrap compiles forward refs through var cells, but keeping
|
|
them to one keeps the compiled namespace simple."
|
|
(:require [jolt.ir :refer [const local var-ref host-ref if-node do-node invoke
|
|
def-node let-node fn-node vector-node map-node set-node
|
|
quote-node throw-node]]
|
|
[jolt.host :refer [form-sym? form-sym-name form-sym-ns form-list?
|
|
form-vec? form-map? form-set? form-char?
|
|
form-literal? form-elements form-vec-items
|
|
form-map-pairs form-set-items form-special? compile-ns
|
|
form-macro? form-expand-1 resolve-global
|
|
form-sym-meta host-intern! form-syntax-quote-lower]]))
|
|
|
|
(declare analyze)
|
|
|
|
(def ^:private handled
|
|
#{"quote" "if" "do" "def" "fn*" "let*" "loop*" "recur" "throw" "try"
|
|
"syntax-quote"})
|
|
|
|
(defn- uncompilable [why]
|
|
(throw (str "jolt/uncompilable: " why)))
|
|
|
|
(def ^:private gensym-counter (atom 0))
|
|
(defn- gen-name [prefix]
|
|
(let [n @gensym-counter]
|
|
(swap! gensym-counter inc)
|
|
(str "_r$" prefix n)))
|
|
|
|
(defn- empty-env [] {:locals #{}})
|
|
(defn- local? [env nm] (contains? (:locals env) nm))
|
|
(defn- add-locals [env names] (update env :locals #(reduce conj % names)))
|
|
(defn- with-recur [env name] (assoc env :recur name))
|
|
|
|
(defn- analyze-seq [ctx forms env]
|
|
(let [v (mapv #(analyze ctx % env) forms)
|
|
n (count v)]
|
|
(cond
|
|
(zero? n) (const nil)
|
|
(= 1 n) (first v)
|
|
:else (do-node (subvec v 0 (dec n)) (peek v)))))
|
|
|
|
(defn- analyze-bindings [ctx bvec env]
|
|
(loop [i 0 env env pairs []]
|
|
(if (< i (count bvec))
|
|
(let [bsym (nth bvec i)]
|
|
(when-not (form-sym? bsym) (uncompilable "destructuring binding"))
|
|
(let [nm (form-sym-name bsym)
|
|
init (analyze ctx (nth bvec (inc i)) env)]
|
|
(recur (+ i 2) (add-locals env [nm]) (conj pairs [nm init]))))
|
|
[pairs env])))
|
|
|
|
(defn- parse-params [pvec]
|
|
(loop [i 0 fixed [] rest-name nil]
|
|
(if (< i (count pvec))
|
|
(let [p (nth pvec i)]
|
|
(when-not (form-sym? p) (uncompilable "destructuring fn param"))
|
|
(if (= "&" (form-sym-name p))
|
|
(let [r (nth pvec (inc i))]
|
|
(when-not (form-sym? r) (uncompilable "destructuring fn rest"))
|
|
(recur (+ i 2) fixed (form-sym-name r)))
|
|
(recur (inc i) (conj fixed (form-sym-name p)) rest-name)))
|
|
{:fixed fixed :rest rest-name})))
|
|
|
|
(defn- analyze-arity [ctx pvec body env fn-name]
|
|
(let [pp (parse-params (vec (form-vec-items pvec)))
|
|
fixed (:fixed pp)
|
|
rst (:rest pp)
|
|
;; Always a recur target, variadic included: the back end gives the rest
|
|
;; param an ordinary positional slot (holding the collected seq), so recur
|
|
;; is a self-call carrying the rest seq directly — Clojure semantics.
|
|
rname (gen-name "arity")
|
|
names (cond-> (vec fixed) rst (conj rst) fn-name (conj fn-name))
|
|
env* (-> (add-locals env names) (with-recur rname))
|
|
arity {:params fixed :recur-name rname
|
|
:body (analyze-seq ctx body env*)}]
|
|
;; :rest only when variadic — an absent :rest reads back nil, same as before,
|
|
;; but keeps a fixed arity a nil-free struct rather than a phm.
|
|
(if rst (assoc arity :rest rst) arity)))
|
|
|
|
(defn- analyze-fn [ctx items env]
|
|
(let [named (form-sym? (nth items 1))
|
|
fn-name (when named (form-sym-name (nth items 1)))
|
|
rest-items (if named (drop 2 items) (drop 1 items))
|
|
first* (first rest-items)]
|
|
(cond
|
|
(form-vec? first*)
|
|
(fn-node fn-name [(analyze-arity ctx first* (rest rest-items) env fn-name)])
|
|
(form-list? first*)
|
|
(fn-node fn-name
|
|
(mapv (fn [clause]
|
|
(let [cl (vec (form-elements clause))]
|
|
(analyze-arity ctx (first cl) (rest cl) env fn-name)))
|
|
rest-items))
|
|
:else (uncompilable "fn: bad params"))))
|
|
|
|
(defn- analyze-try [ctx items env]
|
|
(let [clauses (rest items)
|
|
body (atom [])
|
|
catch-sym (atom nil)
|
|
catch-body (atom nil)
|
|
finally-body (atom nil)]
|
|
(doseq [c clauses]
|
|
(let [head (when (form-list? c) (first (vec (form-elements c))))
|
|
hname (when (and head (form-sym? head)) (form-sym-name head))]
|
|
(cond
|
|
(= hname "catch")
|
|
(let [cl (vec (form-elements c))]
|
|
(reset! catch-sym (form-sym-name (nth cl 2)))
|
|
(reset! catch-body (drop 3 cl)))
|
|
(= hname "finally")
|
|
(reset! finally-body (rest (vec (form-elements c))))
|
|
:else (swap! body conj c))))
|
|
{:op :try
|
|
:body (analyze-seq ctx @body env)
|
|
:catch-sym @catch-sym
|
|
:catch-body (when @catch-body
|
|
(analyze-seq ctx @catch-body (add-locals env [@catch-sym])))
|
|
:finally (when @finally-body (analyze-seq ctx @finally-body env))}))
|
|
|
|
(defn- analyze-special [ctx op items env]
|
|
(case op
|
|
"quote" (quote-node (second items))
|
|
"if" (if-node (analyze ctx (nth items 1) env)
|
|
(analyze ctx (nth items 2) env)
|
|
(if (> (count items) 3)
|
|
(analyze ctx (nth items 3) env)
|
|
(const nil)))
|
|
"do" (analyze-seq ctx (rest items) env)
|
|
"throw" (throw-node (analyze ctx (nth items 1) env))
|
|
"def" (let [name-sym (nth items 1)
|
|
nm (form-sym-name name-sym)
|
|
cur (compile-ns ctx)]
|
|
(host-intern! ctx cur nm)
|
|
(def-node cur nm (analyze ctx (nth items 2) env) (form-sym-meta name-sym)))
|
|
"let*" (let [bvec (vec (form-vec-items (nth items 1)))
|
|
r (analyze-bindings ctx bvec env)]
|
|
(let-node (first r) (analyze-seq ctx (drop 2 items) (second r))))
|
|
"loop*" (let [bvec (vec (form-vec-items (nth items 1)))
|
|
rname (gen-name "loop")
|
|
r (analyze-bindings ctx bvec env)
|
|
env** (with-recur (second r) rname)]
|
|
{:op :loop :recur-name rname :bindings (first r)
|
|
:body (analyze-seq ctx (drop 2 items) env**)})
|
|
"recur" (let [rt (:recur env)]
|
|
(when-not rt (uncompilable "recur outside loop/fn"))
|
|
{:op :recur :recur-name rt
|
|
:args (mapv #(analyze ctx % env) (rest items))})
|
|
"try" (analyze-try ctx items env)
|
|
"fn*" (analyze-fn ctx items env)
|
|
;; Lower the backtick to construction code (zero runtime cost), then analyze
|
|
;; it — the macroexpand/compile-time step, per read -> macroexpand -> compile.
|
|
"syntax-quote" (analyze ctx (form-syntax-quote-lower ctx (second items)) env)
|
|
(uncompilable (str "special form " op))))
|
|
|
|
(defn- analyze-symbol [ctx form env]
|
|
(let [nm (form-sym-name form) ns (form-sym-ns form)]
|
|
(cond
|
|
(and (nil? ns) (local? env nm)) (local nm)
|
|
ns (let [r (resolve-global ctx form)]
|
|
(if (= :var (:kind r))
|
|
(var-ref (:ns r) (:name r))
|
|
(uncompilable (str "qualified ref " ns "/" nm))))
|
|
:else (let [r (resolve-global ctx form)]
|
|
(case (:kind r)
|
|
:var (var-ref (:ns r) (:name r))
|
|
:host (host-ref (:name r))
|
|
(var-ref (compile-ns ctx) nm))))))
|
|
|
|
(defn- analyze-list [ctx form env]
|
|
(let [items (vec (form-elements form))]
|
|
(if (zero? (count items))
|
|
(quote-node form)
|
|
(let [head (first items)
|
|
hname (when (and (form-sym? head) (nil? (form-sym-ns head))) (form-sym-name head))
|
|
shadowed (and hname (local? env hname))]
|
|
(cond
|
|
(and hname (not shadowed) (contains? handled hname))
|
|
(analyze-special ctx hname items env)
|
|
(and hname (not shadowed) (form-special? hname))
|
|
(uncompilable (str "special form " hname))
|
|
(and (form-sym? head) (not shadowed) (form-macro? ctx head))
|
|
(analyze ctx (form-expand-1 ctx form) env)
|
|
:else
|
|
(invoke (analyze ctx head env)
|
|
(mapv #(analyze ctx % env) (rest items))))))))
|
|
|
|
(defn analyze
|
|
([ctx form] (analyze ctx form (empty-env)))
|
|
([ctx form env]
|
|
(cond
|
|
(form-literal? form) (const form)
|
|
(form-sym? form) (analyze-symbol ctx form env)
|
|
(form-vec? form) (vector-node (mapv #(analyze ctx % env) (form-vec-items form)))
|
|
(form-map? form) (map-node (mapv (fn [p] [(analyze ctx (first p) env)
|
|
(analyze ctx (second p) env)])
|
|
(form-map-pairs form)))
|
|
(form-set? form) (set-node (mapv #(analyze ctx % env) (form-set-items form)))
|
|
(form-list? form) (analyze-list ctx form env)
|
|
:else (uncompilable "unsupported form"))))
|