diff --git a/docs/rfc/0004-type-hints.md b/docs/rfc/0004-type-hints.md index 4c84837..5c47070 100644 --- a/docs/rfc/0004-type-hints.md +++ b/docs/rfc/0004-type-hints.md @@ -81,6 +81,33 @@ The same machinery covers both `(:k m)` and `(get m :k [default])` when the key is a constant keyword. A `get` with a variable, numeric, or string key falls through to `core-get` unchanged. +## Record hints across namespaces, and as inference seeds + +A `^RecordType` hint does two things beyond dropping the lookup guard. + +**It carries the specific type, not just "a struct".** The guard-skip only needs +to know the value is raw-get-safe (`:struct`), but the structural inference (RFC +0005) wants the actual record type so a field read gets the field's type — +`(:origin ray)` on a `^Ray ray` is a `Vec3`, not `:any`. A record hint on a +parameter is resolved to the record's constructor key and used to **seed the +inference's parameter type**. That is what keeps a record parameter's reads typed +across a namespace boundary *without* whole-program inference (RFC 0005, +"Cross-namespace inference") — the open-world counterpart to the whole-program +pass. Hinting only the public entry point is not enough; the hint has to be on +the function where the hot reads actually happen. + +**It resolves across namespaces.** A hint may name a record defined in another +namespace, in either spelling — `^Vec3` where the type is `:refer`-ed, or +`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key` in +`src/jolt/host_iface.janet`, backed by `record-hint-ctor-key` in +`src/jolt/evaluator.janet`) runs against the *compile* namespace and maps the +type to its home constructor key through a constructor-value index — keyed by the +constructor value, not a var's namespace, so a `:refer`-interned var (whose +namespace is the referring one) still resolves home. The reader keeps a tag's +namespace qualifier (`^v/Vec3` → `"v/Vec3"`, not `"Vec3"`) so the aliased +spelling has something to resolve. Both `defrecord` field hints and function +parameter hints use this resolution. + ## Soundness and the checked mode An accurate hint is correctness-preserving by construction: for a struct or diff --git a/docs/rfc/0005-structural-type-inference.md b/docs/rfc/0005-structural-type-inference.md index 635e15c..42cad41 100644 --- a/docs/rfc/0005-structural-type-inference.md +++ b/docs/rfc/0005-structural-type-inference.md @@ -163,6 +163,63 @@ wrong specialization is therefore impossible. The inter-procedural part keeps the closed-world (optimization-mode) assumption already adopted, which is sound under whole-program / source-distribution compilation. +## Compilation modes and defaults + +Direct-linking — and the inference and specialization it enables — is the +**default for running a program** and stays **off for interactive work**, chosen +by the CLI run mode rather than a global opt-in flag: + +| mode | linking | whole-program | +|---|---|---| +| `-m` / `-M NS` (program entry) | direct (default) | **auto** (closed world) | +| `FILE` / `-f` / stdin (`-`) | direct (default) | no (per-namespace) | +| `repl`, `-e`, `nrepl-server` | indirect / open | no | + +A program run is a closed world — every namespace is required, then the code +runs to completion — so it direct-links: user code gets inlining, record shapes, +and the inference's specialization. A `-m` / `-M` entry is the exact point where +all requires are done and `-main` is about to run, so the whole-program +cross-namespace pass (below) runs there automatically. Interactive modes stay +open: a REPL, `-e`, and the nREPL server must let you redefine vars — which +direct-linking seals against — so they keep the indirect, live-deref path. + +Env overrides, all winning over the mode default: + +- `JOLT_NO_DIRECT_LINK=1` — force the open/indirect path even for a program run + (runtime redefinition, hot-reload, self-modifying code). +- `JOLT_NO_WHOLE_PROGRAM=1` — keep direct-linking but skip the whole-program + pass (per-namespace inference only). +- `JOLT_DIRECT_LINK=1` — force direct-linking on even in an interactive mode. +- `JOLT_WHOLE_PROGRAM=1` — force the whole-program pass on in any direct-linked + mode. +- `JOLT_NO_SHAPE=1` — disable the record/shape representation under direct-linking. + +What direct-linking gives up is what Clojure's `:direct-linking` and jank's +`-Odirect-call` give up: a direct call embeds its callee, so redefining the +callee is not seen by already-compiled callers. Whole-program additionally +const-links stable vars (data defs, record types, `^:redef`), extending the same +trade. That is why the interactive modes stay open and the opt-outs exist. + +### Cross-namespace inference + +Per-namespace inference (a `FILE` run, or any namespace under +`JOLT_NO_WHOLE_PROGRAM`) types a function's parameters from the call sites it can +see **within that namespace**. A function whose record parameter is supplied by a +caller in *another* namespace is left `:any`, its field reads keep the guard, and +the values derived from it widen — so a decomposed program is markedly slower +than the same code in one namespace (measured at ~3.7× on the ray tracer split +across five namespaces). The information exists in the program; per-namespace +compilation just can't see a caller in a not-yet-loaded namespace. Two ways to +supply it: + +1. **Whole-program** (auto for `-m` / `-M`) runs one closed-world inference + fixpoint over every loaded namespace before `-main`, typing each parameter + from its call sites wherever they live. Namespaces required later (inside + `-main`) fall back to per-namespace inference. +2. **Parameter type hints** (`^RecordType`, RFC 0004) declare the type directly, + so it also works in the open world — REPL, library code that must be fast for + any caller, and hot-reloading servers — where the world cannot be closed. + ## Relationship to Hindley-Milner and soft typing This is HM-shaped with two deliberate departures, which is the textbook diff --git a/docs/self-hosting-compiler.md b/docs/self-hosting-compiler.md index ea21c56..b46b57e 100644 --- a/docs/self-hosting-compiler.md +++ b/docs/self-hosting-compiler.md @@ -141,7 +141,10 @@ coverage incrementally, and de-risks the self-hosting bootstrap. **Live flexibility.** Vars stay first-class cells; compiled code derefs them; `def` updates the root; protocol/multimethod dispatch stays dynamic. Direct -linking is opt-in, never the default, so the REPL is always live. +linking seals a call against redefinition, so the interactive modes — the REPL, +`-e`, the nREPL server — always stay live (indirect). Running a *program* (a +file, `-m`/`-M`) direct-links by default, since it's a closed world; opt back out +with `JOLT_NO_DIRECT_LINK`. (See RFC 0005, "Compilation modes and defaults".) ## A staged path diff --git a/src/jolt/backend.janet b/src/jolt/backend.janet index 157f784..f6064cb 100644 --- a/src/jolt/backend.janet +++ b/src/jolt/backend.janet @@ -787,14 +787,16 @@ (def pv (unless (= "1" (os/getenv "JOLT_NO_IR_PASSES")) (ns-find (ctx-find-ns ctx "jolt.passes") "run-passes"))) # Success-type checking level (RFC 0006). JOLT_TYPE_CHECK wins when set; - # otherwise it defaults to `warn` in direct-link builds — where the - # optimization inference already runs, so checking piggybacks on it for nearly - # free — and stays OFF for plain REPL/dev builds (no inference -> no free ride; - # opt in with JOLT_TYPE_CHECK there). (jolt audit) + # otherwise it defaults to `warn` in a direct-link build, where the inference + # already runs so checking piggybacks for nearly free — EXCEPT when direct- + # linking was auto-enabled by a casual program run (:direct-link-auto?), which + # shouldn't spam type warnings. Stays OFF for plain REPL/dev builds too; opt in + # anywhere with JOLT_TYPE_CHECK. (jolt audit) (def tc (os/getenv "JOLT_TYPE_CHECK")) (def tc-off (or (= tc "off") (= tc "0"))) (def direct-link? (if (get (ctx :env) :inline?) true false)) - (def level (cond tc-off nil tc tc direct-link? "warn" true nil)) + (def auto-dl? (if (get (ctx :env) :direct-link-auto?) true false)) + (def level (cond tc-off nil tc tc (and direct-link? (not auto-dl?)) "warn" true nil)) (def uenv (os/getenv "JOLT_TYPE_CHECK_USER")) (def strict? (and uenv (not= uenv "0") (not= uenv "off") true)) # piggyback: check DURING run-passes' inference (direct-link, the cheap path) diff --git a/src/jolt/evaluator.janet b/src/jolt/evaluator.janet index 47ea5ac..3114516 100644 --- a/src/jolt/evaluator.janet +++ b/src/jolt/evaluator.janet @@ -455,10 +455,13 @@ # (a failure here must not break loading). Hook installed by the api to # avoid an evaluator->backend circular import. (when (get (ctx :env) :inline?) - (if (get (ctx :env) :whole-program?) + (if (and (get (ctx :env) :whole-program?) + (not (get (ctx :env) :infer-program-done?))) # whole-program (jolt-t34): defer — record the ns and run ONE # fixpoint over all units later (the closed-world pass sees every - # caller, so cross-ns param types propagate) + # caller, so cross-ns param types propagate). Once that batch pass + # has run (infer-program-done?), a ns loaded later — a lazy require + # inside -main — can't join it, so fall back to per-ns inference. (let [lst (or (get (ctx :env) :inferred-nses) (let [a @[]] (put (ctx :env) :inferred-nses a) a))] (array/push lst ns-name)) diff --git a/src/jolt/main.janet b/src/jolt/main.janet index c97c56e..bf507b6 100644 --- a/src/jolt/main.janet +++ b/src/jolt/main.janet @@ -433,7 +433,10 @@ # whole-program (jolt-t34): every unit is loaded now — run the one closed- # world fixpoint over all of them before -main, so cross-ns types propagate (when (get (ctx :env) :whole-program?) - (when-let [ip (get (ctx :env) :infer-program!)] (protect (ip ctx)))) + (when-let [ip (get (ctx :env) :infer-program!)] (protect (ip ctx))) + # the batch pass is done — a namespace required later (inside -main) now + # falls back to per-ns inference instead of being deferred and never run. + (put (ctx :env) :infer-program-done? true)) (load-string ctx (string "(apply " ns-name "/-main *command-line-args*)"))) ([err fib] (report-error err fib) (os/exit 1)))) @@ -475,6 +478,14 @@ (print " uberscript OUT -m NS Bundle NS + its required namespaces into one .clj") (print " --version, version Print the Jolt version") (print " -h, --help, help Show this help\n") + (print "Running a program (a file, -m/-M) direct-links and optimizes by default;") + (print "the repl, -e, and nrepl-server stay open so you can redefine vars.") + (print "Environment:") + (print " JOLT_NO_DIRECT_LINK=1 keep a program run open/redefinable (no optimization)") + (print " JOLT_NO_WHOLE_PROGRAM=1 direct-link but skip the cross-namespace pass") + (print " JOLT_DIRECT_LINK=1 force direct-linking on (e.g. for -e)") + (print " JOLT_WHOLE_PROGRAM=1 force the whole-program pass on") + (print " JOLT_INTERPRET=1 run the tree-walking interpreter\n") (print "Dependencies (deps.edn) are handled by the separate jolt-deps tool.")) (def- help-flags {"-h" true "--help" true "help" true "-?" true}) @@ -501,25 +512,51 @@ # nothing. Re-read it here so the env wins in the running process. (when-let [jf (os/getenv "JOLT_FEATURES")] (reader-features-set! (filter |(> (length $) 0) (string/split "," jf)))) - # JOLT_DIRECT_LINK, same story: :direct-linking?/:inline? are baked into ctx at - # build time (init runs during the jpm compile). Re-read here so a running - # process can opt user code into direct-linking + inlining (jolt-87f, the - # AOT-escape-analysis passes). Core is already compiled into the image; this - # only affects user code compiled at runtime. Off by default — user code stays - # fully redefinable unless asked otherwise. - (when (= "1" (os/getenv "JOLT_DIRECT_LINK")) - (put (ctx :env) :direct-linking? true) - (put (ctx :env) :inline? true)) - # Recompute the shape gates from the runtime env (the baked ctx computed them - # at build time, before JOLT_DIRECT_LINK/JOLT_SHAPE were set here). jolt-t34: - # :shapes? = shape-recs active (records), on with direct-linking; :map-shapes? = - # also shape generic maps (opt-in JOLT_SHAPE). + # Linking default depends on the run MODE. Running a PROGRAM (a file, -f, -m/-M, + # stdin) is a closed world — all code is loaded, then it executes to completion + # — so it direct-links by default: user code gets inlining + shapes + the type + # inference's specialization (jolt-87f). INTERACTIVE modes (repl, -e, the nREPL + # server) stay indirect/open so redefinition works — direct-linking would seal + # callers against a redef. Explicit env always wins: JOLT_NO_DIRECT_LINK forces + # the open path even for a program run (runtime redefinition / hot-reload), + # JOLT_DIRECT_LINK forces it on even for -e. Core is already compiled into the + # image; this only governs user code compiled at runtime. + (def open-mode? + (or (empty? argv) + (help-flags (argv 0)) (version-flags (argv 0)) + (= (argv 0) "repl") (nrepl-flags (argv 0)) (eval-flags (argv 0)) + (= (argv 0) "uberscript"))) + (def main-entry? (and (not (empty? argv)) (main-flags (argv 0)))) + (def dl-forced + (cond (os/getenv "JOLT_NO_DIRECT_LINK") :off + (= "1" (os/getenv "JOLT_DIRECT_LINK")) :on + :none)) + (def dl (case dl-forced :off false :on true (not open-mode?))) + (put (ctx :env) :direct-linking? dl) + (put (ctx :env) :inline? dl) + # Mark direct-linking that was AUTO-enabled by the run mode (vs explicitly + # requested). The success checker (RFC 0006) rides on the inference for free, + # but a casual program run shouldn't spam type warnings just because it now + # direct-links — so the checker's default-on is suppressed in the auto case + # (JOLT_TYPE_CHECK still opts in). API/build callers never set this flag, so an + # explicit :direct-linking? there keeps the checker as before. See backend. + (put (ctx :env) :direct-link-auto? (and dl (= dl-forced :none))) + # Shape gates, recomputed from the runtime env (the baked ctx computed them at + # build time). :shapes? = shape-recs active (records), on with direct-linking; + # :map-shapes? = also shape generic maps (opt-in JOLT_SHAPE). (put (ctx :env) :shapes? - (and (get (ctx :env) :direct-linking?) (not (os/getenv "JOLT_NO_SHAPE")))) + (and dl (not (os/getenv "JOLT_NO_SHAPE")))) (put (ctx :env) :map-shapes? (and (os/getenv "JOLT_SHAPE") (not (os/getenv "JOLT_NO_SHAPE")))) + # Whole-program (closed-world cross-namespace inference) auto-enables for a + # -m/-M program entry under direct-linking — that's the point where every + # require is done and -main is about to run. JOLT_WHOLE_PROGRAM forces it on in + # other direct-linked modes; JOLT_NO_WHOLE_PROGRAM opts out. Namespaces required + # later (inside -main) fall back to per-ns inference (see loader). (put (ctx :env) :whole-program? - (and (os/getenv "JOLT_WHOLE_PROGRAM") (get (ctx :env) :direct-linking?))) + (and dl + (not (os/getenv "JOLT_NO_WHOLE_PROGRAM")) + (or main-entry? (os/getenv "JOLT_WHOLE_PROGRAM")))) (cond (empty? argv) (run-repl) (help-flags (argv 0)) (print-help) diff --git a/test/integration/devirt-test.janet b/test/integration/devirt-test.janet index 2304153..190a8e3 100644 --- a/test/integration/devirt-test.janet +++ b/test/integration/devirt-test.janet @@ -25,7 +25,9 @@ " (mapv area [(->Rect 2 3) (->Circ 2)])))\n")) # heterogeneous: [6 12] (def out (string dir "/out.txt")) (def jbin (string (os/cwd) "/" jolt)) - (def cmd (string (if whole? "JOLT_WHOLE_PROGRAM=1 " "") + # -m auto-enables whole-program under direct-linking now, so the per-ns case + # (whole? false) must explicitly opt out to test the dispatched/per-ns path. + (def cmd (string (if whole? "JOLT_WHOLE_PROGRAM=1 " "JOLT_NO_WHOLE_PROGRAM=1 ") "JOLT_DIRECT_LINK=1 JOLT_PATH=" dir " " jbin " -m dv > " out " 2>&1")) (os/execute ["sh" "-c" cmd] :p) (string/trimr (slurp out))) diff --git a/test/integration/whole-program-test.janet b/test/integration/whole-program-test.janet index 3839f65..c711db8 100644 --- a/test/integration/whole-program-test.janet +++ b/test/integration/whole-program-test.janet @@ -39,7 +39,9 @@ (if (not (os/stat jolt)) (print "whole-program: SKIP (no build/jolt — run from source)") - (let [per-ns (run "") + # -m now auto-enables whole-program under direct-linking, so the per-ns + # baseline must explicitly opt out to exercise the per-namespace path. + (let [per-ns (run "JOLT_NO_WHOLE_PROGRAM=1 ") whole (run "JOLT_WHOLE_PROGRAM=1 ")] (printf " per-ns: %s" per-ns) (printf " whole-program: %s" whole)