core: staged-bootstrap kernel tier; move second/peek/subvec/mapv/update to Clojure

First fractal turn of the multi-stage bootstrap (jolt-tzo). The Clojure part of
clojure.core is now loaded in ordered tiers under jolt-core/clojure/core/. The
kernel tier (00-kernel: second/peek/subvec/mapv/update) holds the structural fns
the self-hosted compiler itself uses; in compile mode it is bootstrap-compiled
into clojure.core BEFORE the analyzer is built, so the analyzer binds those names
to the Clojure definitions instead of a not-yet-defined forward ref. That removes
the circularity that previously forced these to stay in Janet — the five core-*
primitives and their init-core! entries are gone.

Mechanism: backend/bootstrap-load-source (generalized from compile-load) builds a
source string into a target ns via the bootstrap; rebuild-compiler! recompiles
the self-hosted compiler against the current core (the rail for future turns,
exercised by the new fixpoint test). api/load-core-overlay! walks the ordered
tiers, bootstrap-loading kernel tiers and self-hosting the rest.

Also fixes a latent evaluator bug surfaced by the move: a fn rebinds current-ns
to its defining ns while it runs and restores on normal return, but a fn that
THREW unwound past its own restore, leaking the ns. try now restores the
try-entry ns on the catch/finally path, so referred-symbol resolution survives a
caught error (this was breaking is/testing in the suite harness after any thrown
assertion). Net battery +3 (3913 -> 3916); conformance 218/218 in all three
modes; binary builds and runs the embedded tiers.

jolt-core/clojure/core.clj

@@ -1,31 +0,0 @@
-;; The Clojure portion of clojure.core. Loaded into the clojure.core namespace at
-;; init (api/init), AFTER the Janet primitives are interned by core/init-core!,
-;; and compiled by the self-hosted pipeline (analyzer -> IR -> Janet back end).
-;;
-;; This is the Phase 4 kernel-shrink seam: fns expressible in plain Clojure on top
-;; of the remaining Janet primitives move here from core.janet, one at a time,
-;; each compiled by the prior stage. Anything here must depend only on core vars
-;; already interned by init-core! (and on other overlay fns defined above it).
-;;
-;; Safe-to-move rule: a fn can move here only if it is (1) NOT in
-;; compiler/core-renames (that map emits core-X Janet symbols directly), (2) has
-;; no internal Janet callers of its core-X binding, and (3) is NOT used by the
-;; self-hosted compiler itself (jolt-core/jolt/*.clj) — the compiler has to
-;; compile this overlay, so anything it calls must already exist as a Janet
-;; primitive. (That last rule is why `second`, used by analyzer.clj, stays in
-;; Janet even though it has no Janet callers.)
-
-(defn ffirst [coll] (first (first coll)))
-(defn nfirst [coll] (next (first coll)))
-(defn fnext  [coll] (first (next coll)))
-(defn nnext  [coll] (next (next coll)))
-
-;; Canonical Clojure defs: pure first/next/loop/recur, no Janet realize-for-iteration.
-(defn last [s]
-  (if (next s) (recur (next s)) (first s)))
-
-(defn butlast [s]
-  (loop [ret [] s s]
-    (if (next s)
-      (recur (conj ret (first s)) (next s))
-      (seq ret))))

jolt-core/clojure/core/00-kernel.clj

@@ -0,0 +1,46 @@
+;; clojure.core — kernel tier (stage just above the Janet seed).
+;;
+;; These are the structural fns the self-hosted compiler itself uses
+;; (jolt.analyzer): second/peek/subvec/mapv/update. Because the compiler must be
+;; able to compile the *rest* of clojure.core, anything it calls has to exist
+;; before it is built. So this tier is loaded FIRST and, in compile mode, is
+;; bootstrap-compiled directly into clojure.core (not routed through the
+;; self-hosted pipeline, which would need these to already exist — the
+;; circularity that previously forced `second` to stay in Janet). With this tier
+;; in place the analyzer is built against the Clojure definitions and the Janet
+;; primitives are gone.
+;;
+;; Constraint: depend only on core-renames primitives (first/next/nth/count/conj/
+;; vec/map/apply/assoc/get/…, all hardwired to the Janet seed) and on each other.
+
+(defn second [coll] (first (next coll)))
+
+(defn peek [coll]
+  (cond
+    (nil? coll) nil
+    ;; vectors (incl. jolt's eager seq results): last element; lists/seqs: first.
+    (vector? coll) (if (zero? (count coll)) nil (nth coll (dec (count coll))))
+    (seq? coll) (first coll)
+    :else (throw (str "peek not supported on: " coll))))
+
+(defn subvec
+  ([v start] (subvec v start (count v)))
+  ([v start end]
+   (when (not (vector? v)) (throw (str "subvec requires a vector")))
+   ;; Clojure coerces indices with (int ...): NaN -> 0, floats/ratios truncate
+   ;; toward zero ((quot x 1)); non-numbers throw. Only then range-check.
+   (let [coerce (fn [x]
+                  (cond
+                    (not (number? x)) (throw (str "subvec index must be a number"))
+                    (not= x x) 0
+                    :else (quot x 1)))
+         s (coerce start)
+         e (coerce end)]
+     (when (or (< s 0) (< e s) (< (count v) e))
+       (throw (str "subvec index out of range: " s " " e)))
+     (loop [i s acc []]
+       (if (< i e) (recur (inc i) (conj acc (nth v i))) acc)))))
+
+(defn mapv [f & colls] (vec (apply map f colls)))
+
+(defn update [m k f & args] (assoc m k (apply f (get m k) args)))

jolt-core/clojure/core/10-seq.clj

@@ -0,0 +1,24 @@
+;; clojure.core — seq tier. Pure-Clojure leaf sequence fns on top of the kernel
+;; tier (00-kernel) and the Janet seed. Loaded after the kernel tier; in compile
+;; mode these self-host through the now-built analyzer (interpreted otherwise).
+;;
+;; Migration rule for adding fns here: the fn must (1) NOT be in
+;; compiler/core-renames (that map emits core-X Janet symbols directly), (2) have
+;; no internal Janet callers of its core-X binding, and (3) NOT be used by the
+;; self-hosted compiler (jolt-core/jolt/*.clj). Compiler-facing structural fns go
+;; in the kernel tier (00-kernel) instead — see its header.
+
+(defn ffirst [coll] (first (first coll)))
+(defn nfirst [coll] (next (first coll)))
+(defn fnext  [coll] (first (next coll)))
+(defn nnext  [coll] (next (next coll)))
+
+;; Canonical Clojure defs: pure first/next/loop/recur, no Janet realize-for-iteration.
+(defn last [s]
+  (if (next s) (recur (next s)) (first s)))
+
+(defn butlast [s]
+  (loop [ret [] s s]
+    (if (next s)
+      (recur (conj ret (first s)) (next s))
+      (seq ret))))

src/jolt/api.janet

@@ -10,6 +10,7 @@
 (use ./compiler)
 (use ./loader)
 (use ./async)
+(import ./backend :as backend)
 (import ./stdlib_embed :as stdlib-embed)
 (import ./host_iface :as host)
 
@@ -27,20 +28,38 @@
     x))
 
 
+# Ordered clojure.core tiers (embedded jolt-core/clojure/core/NN-*.clj). Each tier
+# may reference only the Janet seed + earlier tiers. A :kernel tier holds the
+# structural fns the self-hosted compiler itself uses (second/peek/subvec/mapv/
+# update); in compile mode it must be bootstrap-compiled into clojure.core BEFORE
+# the analyzer is built (the analyzer depends on it), so it bypasses the
+# self-hosted pipeline. Non-kernel tiers route through eval-toplevel like any
+# source (compiled when :compile?, interpreted otherwise — the analyzer, built
+# lazily on the first such form, sees the kernel tier already in place).
+(def- core-tiers
+  [{:ns "clojure.core.00-kernel" :kernel true}
+   {:ns "clojure.core.10-seq"    :kernel false}])
+
+(defn- eval-overlay-source [ctx src]
+  (var s src)
+  (while (> (length (string/trim s)) 0)
+    (def [form rest] (parse-next s))
+    (set s rest)
+    (when (not (nil? form)) (eval-toplevel ctx form))))
+
 (defn- load-core-overlay!
-  "Load the Clojure portion of clojure.core (embedded jolt-core/clojure/core.clj)
+  "Load the Clojure portion of clojure.core in dependency-ordered tiers. See
-  into the clojure.core namespace, routed through eval-toplevel like any other
+  core-tiers and jolt-core/clojure/core/."
-  source (compiled when :compile?, interpreted otherwise)."
   [ctx]
-  (when-let [src (get stdlib-embed/sources "clojure.core")]
+  (def compile? (get (ctx :env) :compile?))
-    (def saved (ctx-current-ns ctx))
+  (def saved (ctx-current-ns ctx))
-    (ctx-set-current-ns ctx "clojure.core")
+  (ctx-set-current-ns ctx "clojure.core")
-    (var s src)
+  (each tier core-tiers
-    (while (> (length (string/trim s)) 0)
+    (when-let [src (get stdlib-embed/sources (tier :ns))]
-      (def [form rest] (parse-next s))
+      (if (and compile? (tier :kernel))
-      (set s rest)
+        (backend/bootstrap-load-source ctx "clojure.core" src)
-      (when (not (nil? form)) (eval-toplevel ctx form)))
+        (eval-overlay-source ctx src))))
-    (ctx-set-current-ns ctx saved)))
+  (ctx-set-current-ns ctx saved))
 
 (defn init
   "Create a new Jolt evaluation context.

src/jolt/backend.janet

@@ -209,32 +209,53 @@
 
 # --- pipeline wiring (the self-hosted compile path) ---
 
-# Compile-load a jolt-core namespace via the bootstrap so it runs as native
+# Bootstrap-compile a source string into target-ns: each form is compiled via the
-# bytecode. The analyzer uses unqualified referred names (jolt.host form-* + the
+# bootstrap (native Janet) compiler and its defs interned in target-ns. This is
-# IR ctors), so the bootstrap's plain :var path compiles it. Stateful forms (the
+# the stage-1 builder — it runs BEFORE the self-hosted analyzer exists, so it's
-# ns/require) fall back to the interpreter. Source from the embedded stdlib map.
+# how both the compiler namespaces (jolt.ir/jolt.analyzer) and the clojure.core
+# kernel tier (the structural fns the analyzer itself calls) get built. The
+# analyzer uses unqualified referred names (jolt.host form-* + IR ctors), so the
+# bootstrap's plain :var path compiles it; stateful forms fall back to interp.
+(defn bootstrap-load-source [ctx target-ns src]
+  (def saved (ctx-current-ns ctx))
+  (ctx-set-current-ns ctx target-ns)
+  (var s src)
+  (while (> (length (string/trim s)) 0)
+    (def parsed (r/parse-next s))
+    (set s (in parsed 1))
+    (def f (in parsed 0))
+    (when (not (nil? f))
+      # Guard BOTH compile and the Janet-compile-of-emitted step: a form whose
+      # emitted Janet is invalid (e.g. a bad splice) falls back to interpreted
+      # definition rather than killing the whole load.
+      (def r (protect (comp/eval-compiled (comp/compile-ast f ctx) ctx)))
+      (unless (r 0) (eval-form ctx @{} f))))
+  (ctx-set-current-ns ctx saved))
+
+# Compile-load an embedded jolt-core namespace by name (source from the stdlib map).
 (defn- compile-load [ctx ns-name]
   (def src (get (get (ctx :env) :embedded-sources @{}) ns-name))
-  (when src
+  (when src (bootstrap-load-source ctx ns-name src)))
-    (def saved (ctx-current-ns ctx))
+
-    (ctx-set-current-ns ctx ns-name)
+# Build the self-hosted compiler (IR ctors + analyzer) via the bootstrap. The
-    (var s src)
+# analyzer's references to clojure.core fns it uses (second/peek/subvec/mapv/
-    (while (> (length (string/trim s)) 0)
+# update) resolve to whatever is interned in clojure.core at this point — so the
-      (def parsed (r/parse-next s))
+# kernel tier must already be loaded (see api/load-core-overlay!).
-      (set s (in parsed 1))
+(defn- build-compiler! [ctx]
-      (def f (in parsed 0))
+  (compile-load ctx "jolt.ir")
-      (when (not (nil? f))
+  (compile-load ctx "jolt.analyzer"))
-        # Guard BOTH compile and the Janet-compile-of-emitted step: a form whose
-        # emitted Janet is invalid (e.g. a bad splice) falls back to interpreted
-        # definition rather than killing the whole load.
-        (def r (protect (comp/eval-compiled (comp/compile-ast f ctx) ctx)))
-        (unless (r 0) (eval-form ctx @{} f))))
-    (ctx-set-current-ns ctx saved)))
 
 (defn- ensure-analyzer [ctx]
   (when (= 0 (length ((ctx-find-ns ctx "jolt.analyzer") :mappings)))
-    (compile-load ctx "jolt.ir")
+    (build-compiler! ctx)))
-    (compile-load ctx "jolt.analyzer")))
+
+(defn rebuild-compiler!
+  "Recompile the self-hosted compiler (jolt.ir + jolt.analyzer) against the
+  CURRENT clojure.core. The fractal turn: once a core tier supplies Clojure
+  definitions the compiler itself uses, rebuilding makes the compiler run on
+  them. Idempotent; re-interns the compiler namespaces over the existing cells."
+  [ctx]
+  (build-compiler! ctx))
 
 (defn analyze-form
   "Run the portable Clojure analyzer (jolt.analyzer/analyze) on a reader form,

src/jolt/core.janet

@@ -939,11 +939,11 @@
           dropped (array/slice c (min n (length c)))]
       (if (jvec? coll) (make-vec dropped) dropped))))))
 
-(defn core-second [coll] (core-first (core-rest coll)))
+# ffirst/nfirst/fnext/nnext/last/butlast (seq tier) and second/peek/subvec/mapv/
-# ffirst / nfirst / fnext / nnext / last / butlast now live in the Clojure overlay
+# update (kernel tier) now live in the Clojure clojure.core tiers under
-# (jolt-core/clojure/core.clj). second stays here — the self-hosted compiler
+# jolt-core/clojure/core/. The kernel tier is bootstrap-compiled before the
-# (analyzer.clj) calls it, so it must exist as a Janet primitive before the
+# self-hosted analyzer is built, so the structural fns the analyzer uses come
-# overlay (which the compiler compiles) loads.
+# from Clojure, not Janet — see api/load-core-overlay! and core/00-kernel.clj.
 
 (defn core-drop-last [a & rest]
   (let [n (if (= 0 (length rest)) 1 a)
@@ -1277,17 +1277,9 @@
     (indexed? m) (do (var i 0) (each x m (set acc (f acc i x)) (++ i))))
   acc)
 
-# peek/pop are defined only on stacks (vectors -> last end, lists -> front);
+# pop is defined only on stacks (vectors -> last end, lists -> front); Clojure
-# Clojure throws on sets/maps/seqs/strings/scalars.
+# throws on sets/maps/seqs/strings/scalars. (peek lives in the Clojure kernel
-(defn core-peek [coll]
+# tier — core/00-kernel.clj.)
-  (cond
-    (nil? coll) nil
-    (plist? coll) (if (pl-empty? coll) nil (pl-first coll))   # list: first
-    (pvec? coll) (if (= 0 (pv-count coll)) nil (pv-nth coll (- (pv-count coll) 1)))  # vector: last
-    (tuple? coll) (if (= 0 (length coll)) nil (in coll (- (length coll) 1)))   # vector: last
-    (array? coll) (if (= 0 (length coll)) nil (in coll 0))   # list: first
-    (error (string "peek not supported on " (type coll)))))
-
 (defn core-pop [coll]
   (cond
     (nil? coll) nil
@@ -1297,22 +1289,7 @@
     (array? coll) (if (= 0 (length coll)) (error "Can't pop empty list") (array/slice coll 1))
     (error (string "pop not supported on " (type coll)))))
 
-# Clojure coerces subvec indices with (int ...): floats truncate and NaN -> 0;
+# subvec lives in the Clojure kernel tier — core/00-kernel.clj.
-# only non-numbers and out-of-range values throw.
-(defn- subvec-idx [x]
-  (cond
-    (not (number? x)) (error "subvec index must be a number")
-    (not= x x) 0           # NaN -> 0
-    (math/trunc x)))
-(defn core-subvec [v start &opt end]
-  (when (not (or (pvec? v) (tuple? v) (array? v)))
-    (error (string "subvec requires a vector, got " (type v))))
-  (let [a (vview v)
-        s (subvec-idx start)
-        e (if (nil? end) (length a) (subvec-idx end))]
-    (when (not (and (>= s 0) (<= s e) (<= e (length a))))
-      (error (string "subvec indices out of range: " s " " e " (length " (length a) ")")))
-    (make-vec (tuple/slice a s e))))
 
 (defn core-trampoline [f & args]
   (var result (apply f args))
@@ -2758,11 +2735,8 @@
   (let [nm (core-get ns :name)]
     (if nm {:jolt/type :symbol :ns nil :name (string nm)} nil)))
 
-# update — works on both structs and tables
+# update lives in the Clojure kernel tier — core/00-kernel.clj. update-in stays
-(defn core-update [m k f & args]
+# (it's recursive and has internal callers).
-  (def f (as-fn f))
-  (core-assoc m k (apply f (core-get m k) args)))
-
 (defn- ks-rest [ks]
   (if (tuple? ks) (tuple/slice ks 1) (array/slice ks 1)))
 
@@ -3102,15 +3076,7 @@
   (let [r @[]] (each x (realize-for-iteration coll) (when (truthy? (pred x)) (array/push r x)))
     (make-vec r)))
 
-(defn core-mapv [f & colls]
+# mapv lives in the Clojure kernel tier — core/00-kernel.clj.
-  (def f (as-fn f))
-  (let [r @[]]
-    (if (= 1 (length colls))
-      (each x (realize-for-iteration (colls 0)) (array/push r (f x)))
-      (let [cs (map realize-for-iteration colls)
-            n (min ;(map length cs))]
-        (var i 0) (while (< i n) (array/push r (apply f (map (fn [c] (in c i)) cs))) (++ i))))
-    (make-vec r)))
 
 (defn- td-interpose [sep]
   (fn [rf]
@@ -3699,9 +3665,7 @@
     "map-indexed" core-map-indexed
     "cycle" core-cycle
     "reduce-kv" core-reduce-kv
-    "peek" core-peek
     "pop" core-pop
-    "subvec" core-subvec
     "trampoline" core-trampoline
     "format" core-format
     "letfn" core-letfn
@@ -3726,7 +3690,6 @@
     "remove" core-remove
     "reduce" core-reduce
     "apply" core-apply
-    "second" core-second
     "doall" core-doall
     "dorun" core-dorun
     "run!" core-run!
@@ -3829,7 +3792,6 @@
     "nthrest" core-nthrest
     "nthnext" core-nthnext
     "filterv" core-filterv
-    "mapv" core-mapv
     "empty" core-empty
     "not-empty" core-not-empty
     "rseq" core-rseq
@@ -4117,7 +4079,6 @@
     "comment" core-comment
     "resolve" core-resolve
     "ns-name" core-ns-name
-    "update" core-update
     "update-in" core-update-in
     "assoc-in" core-assoc-in
     "fnil" core-fnil

src/jolt/evaluator.janet

@@ -929,7 +929,14 @@
               (error {:jolt/type :jolt/exception :value val}))
     "try" (let [body-form (in form 1)
                 clauses (tuple/slice form 2)
-                n (length clauses)]
+                n (length clauses)
+                # current-ns is dynamic state. The interpreter rebinds it to a
+                # fn's defining ns while that fn runs and restores it on normal
+                # return, but a fn that THROWS unwinds past its own restore — so
+                # the ns can leak. try is the unwind boundary: restore the ns that
+                # was current at try entry before running catch/finally, so caught
+                # code (and the harness's is/thrown?) sees the right namespace.
+                try-ns (ctx-current-ns ctx)]
             (var catch-sym nil)
             (var catch-body nil)
             (var finally-body nil)
@@ -952,6 +959,7 @@
               (try
                 (eval-form ctx bindings body-form)
                 ([err]
+                 (ctx-set-current-ns ctx try-ns)
                  (var new-bindings @{})
                  (table/setproto new-bindings bindings)
                  # bind the originally-thrown value (unwrap the :jolt/exception
@@ -974,6 +982,7 @@
                     (run-finally finally-body)
                     result)
                   ([err]
+                   (ctx-set-current-ns ctx try-ns)
                    (run-finally finally-body)
                    (error err)))
                 (eval-form ctx bindings body-form))))

test/integration/clojure-test-suite-test.janet

@@ -25,7 +25,11 @@
 # running thread (Janet OS threads can't be interrupted), so `(deref (future
 # (sleep 1)))` re-raises the unresolved-`Thread/sleep` error — a documented
 # platform gap, not a regression in any previously-working behavior.
-(def baseline-pass 3913)
+# Raised 3913 -> 3916 with the staged-bootstrap kernel tier: the evaluator now
+# restores current-ns when a fn throws across a try boundary (a leaked ns used to
+# break referred-symbol resolution after a caught error), and subvec's index
+# coercion (NaN/float/ratio) is faithful — net +3.
+(def baseline-pass 3916)
 # A file is "clean" when it ran with zero failures AND zero errors.
 (def baseline-clean-files 45)
 # Per-file wall-clock budget (seconds). Normal files finish in well under 1s;

test/integration/staged-bootstrap-test.janet

@@ -0,0 +1,63 @@
+# Staged-bootstrap soundness (jolt-vcx, under epic jolt-tzo).
+#
+# The self-hosted compiler's structural deps — second/peek/subvec/mapv/update —
+# now come from the Clojure kernel tier (jolt-core/clojure/core/00-kernel.clj),
+# bootstrap-compiled into clojure.core BEFORE the analyzer is built. This pins
+# the two properties that make that safe:
+#
+#   1. Compile mode: the analyzer (which itself calls second/peek/subvec/mapv)
+#      compiles analyzer-exercising forms correctly — the exact case that broke
+#      when `second` was a plain overlay fn: (first {:a 1}) / (key (first ...)).
+#   2. Bootstrap FIXPOINT: rebuilding the compiler (rebuild-compiler!) against the
+#      now Clojure-defined core still yields a correct compiler. This is the
+#      soundness gate for every future fractal turn (S2 -> S3).
+
+(use ../../src/jolt/api)
+(import ../../src/jolt/backend :as backend)
+
+(var failures 0)
+
+# Each probe is a jolt boolean expression; compared with jolt's own `=`.
+(def probes
+  ["(= 2 (second [1 2 3]))"
+   "(= nil (second [1]))"
+   "(= 3 (peek [1 2 3]))"
+   "(= 1 (peek (list 1 2 3)))"
+   "(= nil (peek []))"
+   "(= [2 3] (subvec [1 2 3 4 5] 1 3))"
+   "(= [3 4 5] (subvec [1 2 3 4 5] 2))"
+   "(= [2 3 4] (mapv inc [1 2 3]))"
+   "(= [11 22 33] (mapv + [1 2 3] [10 20 30]))"
+   "(= {:a 2} (update {:a 1} :a inc))"
+   "(= {:a 1 :b 1} (update {:a 1} :b (fnil inc 0)))"
+   # Regression: these run the analyzer's own second/map-pair path in compile mode.
+   "(= [:a 1] (first {:a 1}))"
+   "(= :a (key (first {:a 1})))"
+   "(= 1 (val (first {:a 1})))"
+   "(= 3 (let [[a b] [1 2]] (+ a b)))"
+   "(= 3 (loop [i 0 acc 0] (if (< i 3) (recur (inc i) (+ acc i)) acc)))"])
+
+(defn- run-probes [ctx label]
+  (each prog probes
+    (def got (protect (eval-string ctx prog)))
+    (unless (and (got 0) (= (got 1) true))
+      (++ failures)
+      (printf "FAIL [%s] %s => %s" label prog
+              (if (got 0) (string/format "%q" (got 1)) (string "ERR:" (got 1)))))))
+
+# Interpret mode: kernel tier interpreted, no analyzer involved.
+(run-probes (init {}) "interpret")
+
+# Compile mode: kernel tier bootstrap-compiled, analyzer built against it.
+(def cctx (init {:compile? true}))
+(run-probes cctx "compile")
+
+# Fixpoint: rebuild the compiler against the current (Clojure-defined) core and
+# re-run. A correct compiler recompiled on the language it just defined stays
+# correct.
+(backend/rebuild-compiler! cctx)
+(run-probes cctx "compile+rebuilt")
+
+(if (pos? failures)
+  (do (printf "staged-bootstrap: %d failure(s)" failures) (os/exit 1))
+  (print "staged-bootstrap: all probes passed (interpret, compile, compile+rebuilt)"))