Direct-linking for closed-world builds (jolt build)

A release/optimized `jolt build` is a closed world: every app def is final, so
an app->app call can bind to the def's Scheme binding directly instead of going
through (jolt-invoke (var-deref ns name)).

The emitter gains a direct-link mode (off for the seed mint, runtime -e/repl, and
dev builds). With it on, a top-level app def also emits a binding jv$<ns>$<name>
that def-var! aliases; an app->app call or value-ref to a name already emitted in
the unit lowers to that binding, skipping both the var-table lookup and the
generic IFn dispatch. ^:dynamic/^:redef defs and nested defs (a defonce's inner
def) opt out and stay indirect. Off direct-link mode, emit-top-form is exactly
emit, so the seed and runtime eval are byte-unchanged (selfhost holds).

build.ss turns it on for release + optimized; the defined-set accumulates across
the dependency-ordered namespaces so a dep's defs are linkable by the time the
entry that calls them is emitted. App->core calls stay indirect for now (core is
the baked seed); that's a later stage.

~1.74x on a hot cross-namespace call loop (26.5s -> 15.2s).

jolt-core/jolt/backend_scheme.clj

@@ -83,6 +83,31 @@
 (def prelude-mode? (atom false))
 (defn set-prelude-mode! [on] (reset! prelude-mode? on))
 
+;; DIRECT-LINK MODE. Off for ordinary runs, the seed mint, and `-e`/repl/load-string
+;; (open world — vars are redefinable). `jolt build` (release/optimized) flips it on
+;; during app emission: a closed-world program where every app def is final, so an
+;; app->app call binds to the def's Scheme binding directly, skipping the var-table
+;; lookup and the generic jolt-invoke dispatch.
+(def direct-link? (atom false))
+(defn set-direct-link! [on] (reset! direct-link? on))
+
+;; Fully-qualified app var names ("ns/name") already emitted with a direct-link
+;; binding in the current unit. A call/value-ref direct-links only to a name in this
+;; set — one defined earlier in emission order (or itself), so the Scheme binding
+;; exists by the time the reference runs. Reset per build.
+(def direct-link-defined (atom #{}))
+(defn direct-link-reset! [] (reset! direct-link-defined #{}))
+
+;; A direct-link Scheme binding name for a var. The fqn maps to a unique identifier
+;; jv$<ns>$<name>; chars that break a Scheme identifier or the `$` separator are
+;; escaped so distinct vars never collide.
+(defn- dl-munge [s]
+  (-> s (str/replace "$" "_D_") (str/replace "#" "_H_") (str/replace "'" "_Q_")))
+(defn- dl-name [ns nm] (str "jv$" (dl-munge ns) "$" (dl-munge nm)))
+(defn- dl-fqn [ns nm] (str ns "/" nm))
+(defn- direct-linkable? [ns nm]
+  (and @direct-link? (contains? @direct-link-defined (dl-fqn ns nm))))
+
 ;; recur-target and the set of munged local names known to hold a procedure (a
 ;; named fn's self-recursion name) are lexically scoped — dynamic vars so the
 ;; recursion auto-restores them (no manual save/restore, no throw-leak).
@@ -412,6 +437,11 @@
       ;; a :local callee that isn't a known procedure -> dynamic IFn dispatch.
       (and (= :local (:op fnode)) (not (*known-procs* (munge-name (:name fnode)))))
       (invoke)
+      ;; closed-world direct call: the callee var is an app def already emitted with
+      ;; a Scheme binding — call it directly, no var lookup, no jolt-invoke.
+      (and (= :var (:op fnode)) (direct-linkable? (:ns fnode) (:name fnode)))
+      (order-args (fn [as] (str "(" (dl-name (:ns fnode) (:name fnode))
+                                (if (seq as) (str " " (str/join " " as)) "") ")")))
       ;; a late-bound :var call head can hold a procedure OR a non-applicable
       ;; value the RT dispatches (multimethod, keyword/coll IFn) — route via
       ;; jolt-invoke (transparent for a procedure).
@@ -453,6 +483,9 @@
     :var (let [core-proc (and (= "clojure.core" (:ns node)) (core-value-procs (:name node)))]
            (cond
              core-proc core-proc
+             ;; direct-linked app var used as a value -> reference its binding (same
+             ;; root as the var cell for a final var; helps DCE keep it live).
+             (direct-linkable? (:ns node) (:name node)) (dl-name (:ns node) (:name node))
              (and (stdlib-var? node) (not (deref prelude-mode?)))
              (throw (ex-info (str "emit: unsupported stdlib ref `" (:ns node) "/" (:name node)
                                   "` (no core on Chez yet)") {}))
@@ -527,3 +560,32 @@
            (str "(def-var! " (chez-str-lit (:ns node)) " " (chez-str-lit (:name node)) " "
                 (emit (:init node)) ")"))
     (throw (ex-info (str "emit: op not yet ported / unhandled: " (pr-str (:op node))) {}))))
+
+;; ^:dynamic / ^:redef on a def opts it out of direct-linking: it stays redefinable,
+;; so callers must go through the var cell. m is a def's :meta (a jolt map value).
+(defn- dl-opt-out? [m] (or (get m :dynamic) (get m :redef)))
+
+;; Per-form entry used by the image/build emitter. In direct-link mode a TOP-LEVEL
+;; def (form root, or spliced from a top-level do) without an opt-out also binds
+;; jv$<fqn> and aliases the var cell to it, so app->app calls/refs bind directly.
+;; Off direct-link mode this is exactly `emit`, so the seed mint and runtime eval are
+;; byte-unchanged. Nested defs (a defonce's inner def) never reach a top-level branch
+;; here, so they stay indirect — a `define` would be illegal in their position.
+(defn emit-top-form [node]
+  (cond
+    (not @direct-link?) (emit node)
+    ;; top-level do splices: each statement/ret is itself a top-level form.
+    (= :do (:op node))
+    (str "(begin " (str/join " " (map emit-top-form (:statements node)))
+         (if (empty? (:statements node)) "" " ") (emit-top-form (:ret node)) ")")
+    (and (= :def (:op node)) (not (:no-init node)) (not (dl-opt-out? (:meta node))))
+    (let [ns (:ns node) nm (:name node) b (dl-name ns nm)]
+      ;; register before emitting the init so a self-referential body direct-links.
+      (swap! direct-link-defined conj (dl-fqn ns nm))
+      (let [init (emit (:init node))]
+        (if (jmeta-nonempty? (:meta node))
+          (str "(begin (define " b " " init ") (def-var-with-meta! "
+               (chez-str-lit ns) " " (chez-str-lit nm) " " b " " (emit-quoted (:meta node)) "))")
+          (str "(begin (define " b " " init ") (def-var! "
+               (chez-str-lit ns) " " (chez-str-lit nm) " " b "))"))))
+    :else (emit node)))