Merge pull request #100 from jolt-lang/default-direct-link

Direct-link + whole-program by default for program runs

docs/rfc/0004-type-hints.md

@@ -81,6 +81,33 @@ The same machinery covers both `(:k m)` and `(get m :k [default])` when the key
 is a constant keyword. A `get` with a variable, numeric, or string key falls
 through to `core-get` unchanged.
 
+## Record hints across namespaces, and as inference seeds
+
+A `^RecordType` hint does two things beyond dropping the lookup guard.
+
+**It carries the specific type, not just "a struct".** The guard-skip only needs
+to know the value is raw-get-safe (`:struct`), but the structural inference (RFC
+0005) wants the actual record type so a field read gets the field's type —
+`(:origin ray)` on a `^Ray ray` is a `Vec3`, not `:any`. A record hint on a
+parameter is resolved to the record's constructor key and used to **seed the
+inference's parameter type**. That is what keeps a record parameter's reads typed
+across a namespace boundary *without* whole-program inference (RFC 0005,
+"Cross-namespace inference") — the open-world counterpart to the whole-program
+pass. Hinting only the public entry point is not enough; the hint has to be on
+the function where the hot reads actually happen.
+
+**It resolves across namespaces.** A hint may name a record defined in another
+namespace, in either spelling — `^Vec3` where the type is `:refer`-ed, or
+`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key` in
+`src/jolt/host_iface.janet`, backed by `record-hint-ctor-key` in
+`src/jolt/evaluator.janet`) runs against the *compile* namespace and maps the
+type to its home constructor key through a constructor-value index — keyed by the
+constructor value, not a var's namespace, so a `:refer`-interned var (whose
+namespace is the referring one) still resolves home. The reader keeps a tag's
+namespace qualifier (`^v/Vec3` → `"v/Vec3"`, not `"Vec3"`) so the aliased
+spelling has something to resolve. Both `defrecord` field hints and function
+parameter hints use this resolution.
+
 ## Soundness and the checked mode
 
 An accurate hint is correctness-preserving by construction: for a struct or

docs/rfc/0005-structural-type-inference.md

@@ -163,6 +163,63 @@ wrong specialization is therefore impossible. The inter-procedural part keeps
 the closed-world (optimization-mode) assumption already adopted, which is sound
 under whole-program / source-distribution compilation.
 
+## Compilation modes and defaults
+
+Direct-linking — and the inference and specialization it enables — is the
+**default for running a program** and stays **off for interactive work**, chosen
+by the CLI run mode rather than a global opt-in flag:
+
+| mode | linking | whole-program |
+|---|---|---|
+| `-m` / `-M NS` (program entry) | direct (default) | **auto** (closed world) |
+| `FILE` / `-f` / stdin (`-`) | direct (default) | no (per-namespace) |
+| `repl`, `-e`, `nrepl-server` | indirect / open | no |
+
+A program run is a closed world — every namespace is required, then the code
+runs to completion — so it direct-links: user code gets inlining, record shapes,
+and the inference's specialization. A `-m` / `-M` entry is the exact point where
+all requires are done and `-main` is about to run, so the whole-program
+cross-namespace pass (below) runs there automatically. Interactive modes stay
+open: a REPL, `-e`, and the nREPL server must let you redefine vars — which
+direct-linking seals against — so they keep the indirect, live-deref path.
+
+Env overrides, all winning over the mode default:
+
+- `JOLT_NO_DIRECT_LINK=1` — force the open/indirect path even for a program run
+  (runtime redefinition, hot-reload, self-modifying code).
+- `JOLT_NO_WHOLE_PROGRAM=1` — keep direct-linking but skip the whole-program
+  pass (per-namespace inference only).
+- `JOLT_DIRECT_LINK=1` — force direct-linking on even in an interactive mode.
+- `JOLT_WHOLE_PROGRAM=1` — force the whole-program pass on in any direct-linked
+  mode.
+- `JOLT_NO_SHAPE=1` — disable the record/shape representation under direct-linking.
+
+What direct-linking gives up is what Clojure's `:direct-linking` and jank's
+`-Odirect-call` give up: a direct call embeds its callee, so redefining the
+callee is not seen by already-compiled callers. Whole-program additionally
+const-links stable vars (data defs, record types, `^:redef`), extending the same
+trade. That is why the interactive modes stay open and the opt-outs exist.
+
+### Cross-namespace inference
+
+Per-namespace inference (a `FILE` run, or any namespace under
+`JOLT_NO_WHOLE_PROGRAM`) types a function's parameters from the call sites it can
+see **within that namespace**. A function whose record parameter is supplied by a
+caller in *another* namespace is left `:any`, its field reads keep the guard, and
+the values derived from it widen — so a decomposed program is markedly slower
+than the same code in one namespace (measured at ~3.7× on the ray tracer split
+across five namespaces). The information exists in the program; per-namespace
+compilation just can't see a caller in a not-yet-loaded namespace. Two ways to
+supply it:
+
+1. **Whole-program** (auto for `-m` / `-M`) runs one closed-world inference
+   fixpoint over every loaded namespace before `-main`, typing each parameter
+   from its call sites wherever they live. Namespaces required later (inside
+   `-main`) fall back to per-namespace inference.
+2. **Parameter type hints** (`^RecordType`, RFC 0004) declare the type directly,
+   so it also works in the open world — REPL, library code that must be fast for
+   any caller, and hot-reloading servers — where the world cannot be closed.
+
 ## Relationship to Hindley-Milner and soft typing
 
 This is HM-shaped with two deliberate departures, which is the textbook

docs/self-hosting-compiler.md

@@ -141,7 +141,10 @@ coverage incrementally, and de-risks the self-hosting bootstrap.
 
 **Live flexibility.** Vars stay first-class cells; compiled code derefs them;
 `def` updates the root; protocol/multimethod dispatch stays dynamic. Direct
-linking is opt-in, never the default, so the REPL is always live.
+linking seals a call against redefinition, so the interactive modes — the REPL,
+`-e`, the nREPL server — always stay live (indirect). Running a *program* (a
+file, `-m`/`-M`) direct-links by default, since it's a closed world; opt back out
+with `JOLT_NO_DIRECT_LINK`. (See RFC 0005, "Compilation modes and defaults".)
 
 ## A staged path
 

src/jolt/backend.janet

@@ -787,14 +787,16 @@
   (def pv (unless (= "1" (os/getenv "JOLT_NO_IR_PASSES"))
             (ns-find (ctx-find-ns ctx "jolt.passes") "run-passes")))
   # Success-type checking level (RFC 0006). JOLT_TYPE_CHECK wins when set;
-  # otherwise it defaults to `warn` in direct-link builds — where the
-  # optimization inference already runs, so checking piggybacks on it for nearly
-  # free — and stays OFF for plain REPL/dev builds (no inference -> no free ride;
-  # opt in with JOLT_TYPE_CHECK there). (jolt audit)
+  # otherwise it defaults to `warn` in a direct-link build, where the inference
+  # already runs so checking piggybacks for nearly free — EXCEPT when direct-
+  # linking was auto-enabled by a casual program run (:direct-link-auto?), which
+  # shouldn't spam type warnings. Stays OFF for plain REPL/dev builds too; opt in
+  # anywhere with JOLT_TYPE_CHECK. (jolt audit)
   (def tc (os/getenv "JOLT_TYPE_CHECK"))
   (def tc-off (or (= tc "off") (= tc "0")))
   (def direct-link? (if (get (ctx :env) :inline?) true false))
-  (def level (cond tc-off nil tc tc direct-link? "warn" true nil))
+  (def auto-dl? (if (get (ctx :env) :direct-link-auto?) true false))
+  (def level (cond tc-off nil tc tc (and direct-link? (not auto-dl?)) "warn" true nil))
   (def uenv (os/getenv "JOLT_TYPE_CHECK_USER"))
   (def strict? (and uenv (not= uenv "0") (not= uenv "off") true))
   # piggyback: check DURING run-passes' inference (direct-link, the cheap path)

src/jolt/evaluator.janet

@@ -455,10 +455,13 @@
             # (a failure here must not break loading). Hook installed by the api to
             # avoid an evaluator->backend circular import.
             (when (get (ctx :env) :inline?)
-              (if (get (ctx :env) :whole-program?)
+              (if (and (get (ctx :env) :whole-program?)
+                       (not (get (ctx :env) :infer-program-done?)))
                 # whole-program (jolt-t34): defer — record the ns and run ONE
                 # fixpoint over all units later (the closed-world pass sees every
-                # caller, so cross-ns param types propagate)
+                # caller, so cross-ns param types propagate). Once that batch pass
+                # has run (infer-program-done?), a ns loaded later — a lazy require
+                # inside -main — can't join it, so fall back to per-ns inference.
                 (let [lst (or (get (ctx :env) :inferred-nses)
                               (let [a @[]] (put (ctx :env) :inferred-nses a) a))]
                   (array/push lst ns-name))

src/jolt/main.janet

@@ -433,7 +433,10 @@
       # whole-program (jolt-t34): every unit is loaded now — run the one closed-
       # world fixpoint over all of them before -main, so cross-ns types propagate
       (when (get (ctx :env) :whole-program?)
-        (when-let [ip (get (ctx :env) :infer-program!)] (protect (ip ctx))))
+        (when-let [ip (get (ctx :env) :infer-program!)] (protect (ip ctx)))
+        # the batch pass is done — a namespace required later (inside -main) now
+        # falls back to per-ns inference instead of being deferred and never run.
+        (put (ctx :env) :infer-program-done? true))
       (load-string ctx (string "(apply " ns-name "/-main *command-line-args*)")))
     ([err fib] (report-error err fib) (os/exit 1))))
 
@@ -475,6 +478,14 @@
   (print "  uberscript OUT -m NS  Bundle NS + its required namespaces into one .clj")
   (print "  --version, version    Print the Jolt version")
   (print "  -h, --help, help      Show this help\n")
+  (print "Running a program (a file, -m/-M) direct-links and optimizes by default;")
+  (print "the repl, -e, and nrepl-server stay open so you can redefine vars.")
+  (print "Environment:")
+  (print "  JOLT_NO_DIRECT_LINK=1   keep a program run open/redefinable (no optimization)")
+  (print "  JOLT_NO_WHOLE_PROGRAM=1 direct-link but skip the cross-namespace pass")
+  (print "  JOLT_DIRECT_LINK=1      force direct-linking on (e.g. for -e)")
+  (print "  JOLT_WHOLE_PROGRAM=1    force the whole-program pass on")
+  (print "  JOLT_INTERPRET=1        run the tree-walking interpreter\n")
   (print "Dependencies (deps.edn) are handled by the separate jolt-deps tool."))
 
 (def- help-flags    {"-h" true "--help" true "help" true "-?" true})
@@ -501,25 +512,51 @@
   # nothing. Re-read it here so the env wins in the running process.
   (when-let [jf (os/getenv "JOLT_FEATURES")]
     (reader-features-set! (filter |(> (length $) 0) (string/split "," jf))))
-  # JOLT_DIRECT_LINK, same story: :direct-linking?/:inline? are baked into ctx at
-  # build time (init runs during the jpm compile). Re-read here so a running
-  # process can opt user code into direct-linking + inlining (jolt-87f, the
-  # AOT-escape-analysis passes). Core is already compiled into the image; this
-  # only affects user code compiled at runtime. Off by default — user code stays
-  # fully redefinable unless asked otherwise.
-  (when (= "1" (os/getenv "JOLT_DIRECT_LINK"))
-    (put (ctx :env) :direct-linking? true)
-    (put (ctx :env) :inline? true))
-  # Recompute the shape gates from the runtime env (the baked ctx computed them
-  # at build time, before JOLT_DIRECT_LINK/JOLT_SHAPE were set here). jolt-t34:
-  # :shapes? = shape-recs active (records), on with direct-linking; :map-shapes? =
-  # also shape generic maps (opt-in JOLT_SHAPE).
+  # Linking default depends on the run MODE. Running a PROGRAM (a file, -f, -m/-M,
+  # stdin) is a closed world — all code is loaded, then it executes to completion
+  # — so it direct-links by default: user code gets inlining + shapes + the type
+  # inference's specialization (jolt-87f). INTERACTIVE modes (repl, -e, the nREPL
+  # server) stay indirect/open so redefinition works — direct-linking would seal
+  # callers against a redef. Explicit env always wins: JOLT_NO_DIRECT_LINK forces
+  # the open path even for a program run (runtime redefinition / hot-reload),
+  # JOLT_DIRECT_LINK forces it on even for -e. Core is already compiled into the
+  # image; this only governs user code compiled at runtime.
+  (def open-mode?
+    (or (empty? argv)
+        (help-flags (argv 0)) (version-flags (argv 0))
+        (= (argv 0) "repl") (nrepl-flags (argv 0)) (eval-flags (argv 0))
+        (= (argv 0) "uberscript")))
+  (def main-entry? (and (not (empty? argv)) (main-flags (argv 0))))
+  (def dl-forced
+    (cond (os/getenv "JOLT_NO_DIRECT_LINK") :off
+          (= "1" (os/getenv "JOLT_DIRECT_LINK")) :on
+          :none))
+  (def dl (case dl-forced :off false :on true (not open-mode?)))
+  (put (ctx :env) :direct-linking? dl)
+  (put (ctx :env) :inline? dl)
+  # Mark direct-linking that was AUTO-enabled by the run mode (vs explicitly
+  # requested). The success checker (RFC 0006) rides on the inference for free,
+  # but a casual program run shouldn't spam type warnings just because it now
+  # direct-links — so the checker's default-on is suppressed in the auto case
+  # (JOLT_TYPE_CHECK still opts in). API/build callers never set this flag, so an
+  # explicit :direct-linking? there keeps the checker as before. See backend.
+  (put (ctx :env) :direct-link-auto? (and dl (= dl-forced :none)))
+  # Shape gates, recomputed from the runtime env (the baked ctx computed them at
+  # build time). :shapes? = shape-recs active (records), on with direct-linking;
+  # :map-shapes? = also shape generic maps (opt-in JOLT_SHAPE).
   (put (ctx :env) :shapes?
-    (and (get (ctx :env) :direct-linking?) (not (os/getenv "JOLT_NO_SHAPE"))))
+    (and dl (not (os/getenv "JOLT_NO_SHAPE"))))
   (put (ctx :env) :map-shapes?
     (and (os/getenv "JOLT_SHAPE") (not (os/getenv "JOLT_NO_SHAPE"))))
+  # Whole-program (closed-world cross-namespace inference) auto-enables for a
+  # -m/-M program entry under direct-linking — that's the point where every
+  # require is done and -main is about to run. JOLT_WHOLE_PROGRAM forces it on in
+  # other direct-linked modes; JOLT_NO_WHOLE_PROGRAM opts out. Namespaces required
+  # later (inside -main) fall back to per-ns inference (see loader).
   (put (ctx :env) :whole-program?
-    (and (os/getenv "JOLT_WHOLE_PROGRAM") (get (ctx :env) :direct-linking?)))
+    (and dl
+         (not (os/getenv "JOLT_NO_WHOLE_PROGRAM"))
+         (or main-entry? (os/getenv "JOLT_WHOLE_PROGRAM"))))
   (cond
     (empty? argv) (run-repl)
     (help-flags (argv 0)) (print-help)

test/integration/devirt-test.janet

@@ -25,7 +25,9 @@
          "           (mapv area [(->Rect 2 3) (->Circ 2)])))\n"))       # heterogeneous: [6 12]
   (def out (string dir "/out.txt"))
   (def jbin (string (os/cwd) "/" jolt))
-  (def cmd (string (if whole? "JOLT_WHOLE_PROGRAM=1 " "")
+  # -m auto-enables whole-program under direct-linking now, so the per-ns case
+  # (whole? false) must explicitly opt out to test the dispatched/per-ns path.
+  (def cmd (string (if whole? "JOLT_WHOLE_PROGRAM=1 " "JOLT_NO_WHOLE_PROGRAM=1 ")
                    "JOLT_DIRECT_LINK=1 JOLT_PATH=" dir " " jbin " -m dv > " out " 2>&1"))
   (os/execute ["sh" "-c" cmd] :p)
   (string/trimr (slurp out)))

test/integration/whole-program-test.janet

@@ -39,7 +39,9 @@
 
 (if (not (os/stat jolt))
   (print "whole-program: SKIP (no build/jolt — run from source)")
-  (let [per-ns (run "")
+  # -m now auto-enables whole-program under direct-linking, so the per-ns
+  # baseline must explicitly opt out to exercise the per-namespace path.
+  (let [per-ns (run "JOLT_NO_WHOLE_PROGRAM=1 ")
         whole  (run "JOLT_WHOLE_PROGRAM=1 ")]
     (printf "  per-ns:        %s" per-ns)
     (printf "  whole-program: %s" whole)