These three were the last eager transformers, blocked by jolt-r81: a self-
recursive lazy-seq in the overlay leaks its macro expansion under :compile? when
recursion goes through a top-level name or (fn name …) self-name. Rewriting the
recursion as letfn-bound (the form partition-by/mapcat/dedupe already use, which
compiles cleanly) sidesteps the bug. All three are now lazy in interpret,
compile, and self-host — completing Option A for every transformer.
interleave: canonical lazy cons-recursion (2-arity) + map/concat (n-arity).
reductions: letfn step accumulator. tree-seq: letfn walk + lazy mapcat.
Gate: conformance 246x3, lazy-infinite 40/40 (+interleave/reductions/tree-seq
infinite cases), fixpoint, self-host, specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Option A makes `take` lazy, so the §6.3 counter tests must force the take
result (dorun) to drive realization — an unconsumed take correctly realizes
nothing. With that, all 37 lazy-infinite cases pass and the minimal-realization
counts match (proving the Option A transformers realize exactly the demanded
prefix).
interleave kept eager: a lazy (cons-recursive) version leaks its lazy-seq macro
expansion under :compile? — the same jolt-r81 bug that blocks lazy reductions/
tree-seq. Documented in 20-coll.clj.
Gate: conformance 246x3, lazy-infinite 37/37, fixpoint, self-host, specs+unit
green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Converts the rest of the lazy transformers to return a LazySeq even over a
concrete collection (dropping the eager "preserve representation" branch, which
returned a vector over vector input): drop, distinct, partition, partition-all,
map-indexed, keep, keep-indexed, take-nth, interpose. Each collapses to its
existing lazy branch run over (lazy-from coll); transducer arities unchanged.
Fixes a latent nth bug exposed by this: core-nth's lazy branch walked with
(ls-first cur) truthiness as the end-of-seq test, so a legitimate false/nil
element was mistaken for the end — (nth (map identity [false 1 2]) 0) threw
instead of returning false, and cond-> (whose macro nths over a now-lazy
(drop 2 clauses) containing the boolean clause tests) failed. nth now walks
with seq-done? and reads via core-first.
Gate: conformance 246x3 (+7 cases), lazy-infinite 18/18, fixpoint, self-host,
all specs+unit green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lazy transformers now return a LazySeq even over a concrete vector, matching
Clojure: (seq? (map inc [1 2 3])) is true, (vector? ...) false. Replaces the
"preserve representation" eager branch (which returned a vector over vector
input) by routing concrete colls through lazy-from + the lazy step machinery.
Flipping these surfaced four boundary bugs, all fixed here:
- cons over a lazy-seq returned a raw cell @[x thunk]; a cons-of-a-cons then
treated it as a plain 2-array and leaked the rest-thunk as an element (broke
interleave). cons over lazy now returns a proper LazySeq.
- coll->cells mistook a user vector whose 2nd elem is a function ([first last]
from juxt) for a cons cell. Cons cells are mutable arrays; user data is
immutable — route pvec/plist/tuple through immutable tuples and apply the
[val,fn] cell heuristic only to mutable arrays. Also coerce set/map/string/
buffer via core-seq.
- ~@ splice over a lazy map result iterated a LazySeq as a Janet table (broke
lazy-cat / self-ref fib). syntax-quote* now realizes via d-realize before
splicing; core-sqcat (self-host) already realized.
- core-next did (length r) on a lazy rest (never 0 on a table) and ls-rest
could return nil → (length nil) crash. core-rest never returns nil; core-next
uses seq-done? (realizes one cell). seq-done? moved above core-rest.
normalize-pvecs (test helper) realizes lazy-seqs so Janet-= comparisons work.
Gate: conformance 239x3 (interpret/compile/self-host, +10 Option A cases),
lazy-infinite 18/18, fixpoint, self-host, all specs+unit green. (sci-bootstrap
and clojure-test-suite skip — vendored dirs absent in this checkout.)
Remaining for full Option A consistency (jolt-7w4): drop/map-indexed/keep/
keep-indexed/take-nth/interpose/distinct/partition/partition-all still eager
over concrete input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Added 16 atom-counter laziness tests to lazy-infinite-test.janet:
map, filter, remove, take-while, drop-while, distinct, take-nth,
map-indexed, keep, keep-indexed, interpose, partition, partition-all,
mapcat, dedupe, repeated-inc
Each test wraps input elements in (swap! c inc) and proves only the
exact minimal number of elements are realized. 37/37 pass, 0 timeouts.
Updated phase-5.md §7: "22/22" → "21/21" (interleave commented out),
§6.3 marked Done.
Final gate results:
lazy-infinite: 37/37 (21 value + 16 counter)
conformance: 229/229 ×3
specs: 32/32 files, 0 failures
clojure-test-suite: 3971 pass (↑45), 6 timeouts (↓3)
core-bench: TOTAL 2531 ms (no Phase-4 baseline for comparison)
Root cause: lazy interleave in 20-coll.clj uses lazy-seq macro which
expands to (make-lazy-seq (fn* [] (coll->cells ...))) in compile mode.
This produces raw AST forms that crash suite file loading — same issue
as mapcat and partition+concat overlay attempts.
Fix: revert interleave to eager (vec-based loop). dedupe stays lazy
(uses make-lazy-seq directly, not lazy-seq macro). xml-seq stays
(uses tree-seq which is eager in overlay).
Suite: 3971 pass (up from 3926), 6 timeouts (down from 9), 4628 assertions.
Lazy-infinite: 21/21 (interleave infinite case commented out).
Conformance: 229x3.
Fix & rest destructuring (evaluator.janet): when the original
value is a lazy-seq, derive & rest by walking ls-rest vi steps
instead of slicing the eagerly-realized array from d-realize.
Rewrite core-mapcat as a lazy state machine (core.janet): step
through input colls one element at a time, call f on each tuple,
then yield from f's result collection. No apply-forcing —
all input colls are walked lazily via lazy-from + seq-done?.
Add mapcat to core-renames (compiler.janet) and remove from
Clojure overlay (10-seq.clj) — compile mode now emits direct
calls to the lazy core-mapcat, fixing the pre-existing
protocol-on-record compile error.
Tests: re-enabled mapcat infinite harness case, added 2 & rest
laziness cases. 21/21 lazy-infinite, 229x3 conformance, 32/32 specs.
core-drop-while (core.janet): return cell via realize-ls instead
of raw LazySeq. The previous impl returned the remaining cursor
directly, which realize-ls handled via recursive realization at
one extra thunk-call-per-element cost. Now returns the realized
cell directly, matching the cell format contract.
mapcat (10-seq.clj): revert to standard (apply concat (apply map
f colls)). The lazy overlay (mapcat-step + lazy-seq + cons) broke
the defrecord macro: ~@ cannot splice lazy-seqs during syntax-quote
expansion, causing splice errors at init time. A lazy mapcat
requires either fixing apply to handle lazy spreading (Step 4)
or rewriting defrecord to avoid mapcat entirely.
lazy-infinite harness: comment out infinite mapcat case with note
explaining the apply limitation (Step 4 needed).
Gates verified:
conformance 229/229 x3 (interpret/compile/self-host)
lazy-infinite 18/18
specs 32/32 files, 0 failures
mapcat in 10-seq.clj now uses a defn- mapcat-step helper that walks
lazy map results element-by-element (cons + lazy-seq), with explicit
arities for 1/2/3/4+ colls. The 4+-coll arity still uses apply to
spread colls to map, but apply no longer forces the inner map result.
Previously (apply concat (apply map f colls)) forced the lazy map
result through core-apply realize-for-iteration, which hung on
infinite inputs. The new step-based impl walks one element at a time
via first/rest/seq primitives.
Re-enabled deferred mapcat infinite-input harness case. 19/19 pass.
Gates: lazy-infinite 19/19, conformance 229/229 interpret+self-host
(compile 228/229 — 1 pre-existing protocol error), specs 32/32.
reduce-kv is pure over reduce/keys/get/nth, so it moves to 20-coll with no
host surface. Both branches fold through reduce, which fixes two latent bugs
in the Janet version: on a vector it saw the pvec as a table and folded over
its internal keys instead of the indices, and it ignored reduced entirely.
nil folds to init. Added vector-index, reduced, and nil spec cases plus a
3-mode conformance case for the vector fix.
Spec suites (interpret) for the overlay-migrated fns and both fixes:
- control-flow: if-let/when-let/if-some/when-some else-scope (9 cases)
- predicates: not-any?/idents/numeric preds/NaN?/abs/object?/... (24 cases)
- sequences: nthrest()/nthnext/distinct? value-eq/replace nil/take-last/... (23)
Plus 10 cases added to the all-3-modes conformance set for the highest-risk
fixes (if-let scope, nthrest (), distinct? value-eq), so compile + self-host are
guarded too: conformance 218 -> 228 in all three modes.
Full suite green.
Add a maps-spec defspec covering nil values through the reader (literals), the
construction path, and the op surface (assoc/merge/merge-with/into/conj/zipmap/
select-keys/get-in/dissoc/reduce-kv, nil keys). Raise the clojure-test-suite
baseline 3919 -> 3926 to guard the assertions the fix unlocked.
The reader yields set literals as a form ({:jolt/type :jolt/set ...}); edn now
walks the parsed result and builds actual sets (recursing into maps/vectors;
lists stay lists, never evaluated). Untagged-struct guard so symbols/chars/tagged
literals pass through. 47 -> 49 in the battery; only #uuid (no uuid type) remains.
clojure.edn/read-string now delegates to clojure.core/read-string (the
reader-based special form — parses, never evals) via a private helper, with the
opts-map :eof arity and nil/blank-input handling. 43 -> 47 in the stdlib battery.
This required an evaluator fix: special forms were matched by name only, so
clojure.edn/read-string dispatched the core read-string SPECIAL FORM instead of
the var. Now a head qualified to a non-core namespace resolves to its var; only
unqualified or clojure.core-qualified heads are special forms. Conformance
218/218 all modes; full suite green.
Remaining edn gaps (jolt-b7y): set literals read as a set FORM not a constructed
set, and #uuid has no real type.
clojure.data: ported from the ClojureScript impl (jolt-native equality-partition
fn instead of host-type protocol dispatch). Verified against Clojure's own
data-test (12/13 canonical; the 13th needs nil-valued-map support, jolt-c7h).
Used mapv over the reference's (doall (map …)) — jolt's lazy multi-coll map +
doall don't force as a reduce accumulator (jolt-dzh).
clojure.zip: jolt's was a broken custom reimplementation; replaced with a port of
real clojure.zip (metadata-based locs). Uses (nth loc …) instead of (loc …)
because meta-bearing vectors aren't invocable as fns (jolt-vh5).
core: with-meta now accepts nil metadata (Clojure allows (with-meta x nil)); it
crashed calling keys on nil. This is what unblocked zip's make-node.
New vendored battery (test/clojure-stdlib/, from clojurust's suite with fixtures
corrected to match real Clojure): walk 34, zip 33, data 61 all clean; edn guarded
at 43 (still a stub — jolt-b7y). Conformance 218/218 all modes; full suite green.
The expensive multimethod path is the hierarchy fallback: when a dispatch value
isn't a direct method key, mm-fn walks every key with isa? (derive-based
dispatch). That resolution is now cached per dispatch value on the multimethod
var (:jolt/dispatch-cache), cleared in place by defmethod/prefer-method/
remove-method/remove-all-methods so a redefinition can't be served stale. Direct
(get methods dv) hits stay uncached — already a single lookup.
Test cases added to dispatch-cache-test (compile, interpret, aot-core off):
hierarchy hit is cached, a newly-added specific method is seen, removal re-exposes
the fallback. Conformance 218/218; full suite green.
Soundness gate for self-hosting (jolt-d0r). stage1 = analyzer built by the Janet
bootstrap; stage2 = analyzer rebuilt by compiling jolt.ir + jolt.analyzer through
stage1 (self-host); stage3 = the same self-rebuild again. A corpus of programs
must produce identical results across all three.
Compared BEHAVIORALLY rather than by emitted code: emitted Janet forms embed live
setter/getter closures (identity varies per compile) and the IR carries
representation-level gensyms and pvec internals, so structural equality fights the
representation. Behavioral parity is the property that actually matters and is
representation-independent.
Also corrects the battery baseline 3920 -> 3919: 3920 was a lucky run; 3919 is the
stable value (the 9 timeout-prone tests can shift the count by one).
Three pure function combinators into the collection tier, as compositions of
mapv/every?/some/apply. Three more core-* primitives + table entries gone.
Battery 3916 -> 3920 (the canonical defs are more correct than the prior Janet
ones — some-fn returns the matching value, every-pred is properly boolean).
Conformance 218/218 all modes.
Host-value protocol dispatch (a protocol extended to Number/String/… called on a
host value) recomputed the candidate type-tag list and walked the registry — up
to ~15 map lookups plus an array alloc — on every call. It now caches
(most-specific-host-tag, protocol, method) -> fn, guarded by a registry
generation that bumps in register-protocol-method; re-extending a protocol
invalidates the cache. The single-lookup fast paths (deftype protocol dispatch,
direct multimethod dispatch) are untouched — they're already cheap, so no cache
overhead there. The multimethod hierarchy-fallback cache (narrower: only
derive-based dispatch) is deferred.
Test in test/integration/dispatch-cache-test.janet covers correctness under
re-extension in all three modes. Conformance 218/218; battery 3916.
Calls compile direct (target value embedded) iff the compiling unit has
direct-linking on, the target isn't ^:redef/^:dynamic, and it's an
already-defined fn; otherwise indirect (live var deref, redefinable). Direct
emission is guarded to function values only — embedding a non-function would make
Janet evaluate it as code.
:aot-core? (init opt, default true; JOLT_AOT_CORE=0 disables) compiles the core
tiers with direct-linking on so core->core calls are direct. User/REPL code
compiles indirect by default, so redefining functions in the REPL works with no
annotation, exactly like Clojure. :aot-core? false makes core indirect too — the
whole language becomes redefinable. ^:redef / ^:dynamic on a target force it
indirect even inside a direct-linked unit.
Also fixes a pre-existing gap this surfaced: compiled def dropped ALL var
metadata (so ^:dynamic was silently lost under compilation, and ^:redef couldn't
work). def metadata now flows analyzer -> IR -> back end (new form-sym-meta host
fn, def-node :meta, var-setter-meta) and is applied to the cell.
Matrix test in test/integration/direct-linking-test.janet. Conformance 218/218
in all three modes under both :aot-core? true and false; battery holds 3916;
binary rebuilt.
First fractal turn of the multi-stage bootstrap (jolt-tzo). The Clojure part of
clojure.core is now loaded in ordered tiers under jolt-core/clojure/core/. The
kernel tier (00-kernel: second/peek/subvec/mapv/update) holds the structural fns
the self-hosted compiler itself uses; in compile mode it is bootstrap-compiled
into clojure.core BEFORE the analyzer is built, so the analyzer binds those names
to the Clojure definitions instead of a not-yet-defined forward ref. That removes
the circularity that previously forced these to stay in Janet — the five core-*
primitives and their init-core! entries are gone.
Mechanism: backend/bootstrap-load-source (generalized from compile-load) builds a
source string into a target ns via the bootstrap; rebuild-compiler! recompiles
the self-hosted compiler against the current core (the rail for future turns,
exercised by the new fixpoint test). api/load-core-overlay! walks the ordered
tiers, bootstrap-loading kernel tiers and self-hosting the rest.
Also fixes a latent evaluator bug surfaced by the move: a fn rebinds current-ns
to its defining ns while it runs and restores on normal return, but a fn that
THREW unwound past its own restore, leaking the ns. try now restores the
try-entry ns on the catch/finally path, so referred-symbol resolution survives a
caught error (this was breaking is/testing in the suite harness after any thrown
assertion). Net battery +3 (3913 -> 3916); conformance 218/218 in all three
modes; binary builds and runs the embedded tiers.
Phase 4 kernel-shrink seam. jolt-core/clojure/core.clj holds the Clojure portion
of clojure.core; api/init loads it into the clojure.core namespace after
init-core! interns the Janet primitives, routed through eval-toplevel so it
compiles via the self-hosted pipeline (or interprets when :compile? is off).
First three fns moved off the Janet side: ffirst, fnext, nnext (leaf, no internal
callers). Same results compiled or interpreted; full suite green; battery holds
3913.
eval-toplevel now routes every compiled form through backend/compile-and-eval
(analyzer -> IR -> Janet back end). The bootstrap compiler (compile-ast) stays
only to bootstrap jolt.ir/jolt.analyzer in backend/compile-load. Dropped the
:selfhost? flag and JOLT_SELFHOST env plumbing — self-host is the default.
Three correctness fixes the full-suite-under-self-host sweep exposed:
- analyzer/back end: recur into a variadic arity now compiles. The arity fn
gets an ordinary positional rest param (the collected seq), with a dispatch
wrapper, so (recur fixed... rest-seq) is a clean self-call like Clojure --
instead of punting to the interpreter, which hangs on this (jolt-4df).
- host_iface: interop heads (.foo/.-field/Foo.) and ns-management forms
(all-ns, create-ns, resolve, ...) now mark uncompilable so they fall back
instead of compiling to a bad call.
- back end: named-fn self-reference ((fn self [n] ... (self ...))) binds the
fn name via a var.
Full unit+integration suite green; clojure-test-suite battery holds 3913.
conformance-test now runs every case under interpret, the bootstrap compiler, AND
the self-hosted pipeline (portable Clojure analyzer -> IR -> Janet back end). All
three pass 218/218, so the self-hosted compiler can't silently regress in CI.
suite-worker honors JOLT_SELFHOST=1 to route the clojure-test-suite through the
self-hosted pipeline (for validating once the analyzer compiles).
The analyzer stays interpreted: compiling it via the bootstrap miscompiles
(qualified refs regressed compile mode 3932->1181; compile-loading yields
corrupted IR) — tracked in jolt-4xc. Correctness is unaffected (the self-hosted
pipeline passes 218/218 conformance); only speed is pending.
Analyzer threads an env (locals + recur target) instead of a bare locals set, and
compiles loop*/recur, recur directly in a fixed-arity fn (each arity's name is its
recur target -> self-call), and try/catch/finally (-> Janet try + defer). recur
into a variadic arity isn't a recur target, so it falls back to the interpreter
rather than miscompiling the rest seq. Still 218/218 conformance via the pipeline,
now with these forms compiling natively instead of interpreting.
The Clojure-in-Clojure front end now compiles and runs a real subset end to end:
arithmetic, if/do/let, vector/map literals, def + name-based global refs, fn,
defn (via macroexpand), recursion through the var cell, multi-arity, variadic,
and higher-order calls. Proven by test/integration/self-host-test.janet.
Pieces:
- jolt-core/jolt/analyzer.clj — PORTABLE Clojure analyzer: reader form -> IR,
depending only on the host contract (jolt.host), never on Janet.
- src/jolt/host_iface.janet — Janet impl of the jolt.host contract (form
introspection + resolve/macro/current-ns), installed into each ctx.
- src/jolt/backend.janet — Janet back end: IR -> Janet form -> eval; resolves
name-based :var nodes to cells and reuses runtime helpers.
Host Janet code lives in src/jolt/ (not a host/janet/ dir): Janet resolves
relative imports per file, so cross-dir ../../src/jolt/* imports load second
instances of compiler/types/core and corrupt state. The portability boundary is
the jolt.host namespace contract + jolt-core/, not the directory.
Notes: gensym in the back end must not be Janet's (shadowed by Jolt's via use);
a jolt bug — (into #{} coll) doesn't add elements — was worked around with reduce
conj in the analyzer (filed separately).
Adds src/jolt/aot.janet: save-ns / load-ns-image marshal a namespace's compiled
var cells to a Janet bytecode image and load them back, skipping
parse/analyze/emit/compile on reload.
Compiled functions close over core fns (cfunctions) that can't marshal by value,
so we marshal against a dictionary built from the baked-in runtime env
(env-lookup jolt-runtime-env, which chains to the Janet boot env): core fns and
builtins are referenced by name, only user bytecode and its var cells are
serialized. The runtime env is identical at save and load time (same binary), so
the dictionaries match.
Test round-trips a namespace (constant, fn over it, core-fn user, recursion) into
a fresh context and confirms the loaded vars run and stay redefinable.
analyze-form now mirrors resolve-sym's full resolution order: jolt var (current
ns / clojure.core), then the runtime/Janet env, then a pending forward-ref cell.
Previously a name that resolves only via the Janet-env fallback (int?, type, …)
interned an empty var and miscompiled; now it emits the runtime binding directly,
matching the interpreter.
suite-worker honors JOLT_COMPILE=1 to run the whole clojure-test-suite through
the compile path. Under compilation it now passes 3932 / errors 124 — at parity
with the interpreter baseline (3913 / 125) across 4600+ assertions, confirming
the hybrid compile path doesn't diverge from the interpreter.
Running the conformance suite under compile mode surfaced many forms that
silently miscompiled (the hybrid fallback only catches compile-time errors,
not wrong results). Fixes:
- Global var resolution now mirrors the interpreter's resolve-var: current ns
(which holds refers) then clojure.core, instead of interning an empty var in
the current ns. This was the big one — every core fn not in core-renames
(iterate, update, subvec, reductions, ...) derefed to nil from a user ns.
- Map literals evaluate their keys and values (were emitted as quoted data);
build via build-map-literal, mirroring the interpreter (struct, or phm when a
key is a collection).
- Vector literals build a mode-appropriate jolt vector via make-vec (pvec when
immutable, array when mutable) instead of a bare Janet tuple, so compiled and
interpreted vectors share one representation (type-strict ops like rseq
rejected tuples).
- Core fn values resolve dynamically from the runtime env instead of a
hand-maintained table that had drifted: core-apply mapped to Janet's native
apply (rejects pvec tails), core-some mapped to core-some?. Removed the table
and the bogus some rename.
- analyze-form throws uncompilable on interpreter special forms it doesn't
implement and on definitional/host macros (deftype, defprotocol, reify,
binding, letfn, read-string, regex/tagged literals, ...), so they fall back
to the interpreter instead of miscompiling — including nested in compiled
forms.
conformance-test.janet now runs every case under both interpreter and compiler
so compile-mode correctness is guarded in CI.
Rewrites fn* analysis/emit. Each arity compiles to a named Janet fn whose
name is its recur target, so recur is a self-call (Janet tail-calls it) —
this fixes recur used directly inside a fn, which previously miscompiled to
a runtime error that the hybrid fallback couldn't catch.
Multi-arity fns dispatch on arg count: fixed arities match exactly, the
variadic arity matches >= its fixed count. The rest param is an ordinary
param holding a seq, so (recur fixed... rest-seq) into a variadic arity
works as in Clojure. Single fixed-arity fns keep the direct, dispatch-free
shape so the hot path stays fast.
Destructuring params still route to the interpreter via uncompilable.
Tests cover named recursion, multi-arity (incl. variadic clause), recur in
fn, and recur into a variadic arity.
eval-one now tries to compile each form and falls back to the interpreter
for anything the compiler can't handle, instead of erroring or silently
miscompiling. Only the compile step is guarded, so runtime errors in
compiled code still propagate (no double-eval of side effects, no hidden
errors).
The compiler now throws jolt/uncompilable on the forms it would otherwise
miscompile — destructuring let/loop bindings and fn params, multi-arity
fns, and named fns — so they route to the interpreter and produce correct
results. These become compile targets in a later optimization pass.
Adds eval-compiled to split compile from eval; compile-mode tests cover
destructuring, multi-arity, named-fn recursion, and error propagation.
Compiled global references now deref through the Jolt var cell instead of an
early-bound Janet symbol, so redefining a def/defn is visible to already-compiled
callers (Janet early-binds plain symbols). A global ref analyzes to :op :var
(resolving/creating the cell); a def sets that same cell's root. Because Janet
copies table constants but references functions, we embed memoized getter/setter
closures over the cell rather than the cell itself. This also subsumes the old
named-fn recursion rewrite and the separate ns-intern.
ns-intern now stores the namespace *name* (string) in a var, not the ns table, so
var cells aren't cyclic (required for embedding).
fib(30) compiled ~0.5s (the late-binding cost vs phase 2's direct-linked 0.08s;
direct-linking for hot calls is a later optimization). Suite green; cross-form,
recursion, and context isolation preserved; redefinition now works.
The Clojure stdlib namespaces (clojure.string/set/walk/edn/zip, jolt.http/
interop/shell/nrepl) were loaded cwd-relative, so the built binary couldn't
load them outside the repo. Now stdlib_embed slurps them all at build time into
a {ns -> source} map frozen in the image, and the loader falls back to it when a
namespace isn't found on disk. The artifact is self-contained — clojure.string
works from any directory.
Drops the one-off nREPL source embed (jolt.nrepl is covered by this now). The
baked-in stdlib is excluded from uberscript bundles since it's part of the
runtime. clojure.core is unchanged (Janet, auto-referred).
Test: embedded-stdlib-test requires from the embedded copy with FS roots cleared.
Phase 3 — bundling. `jolt uberscript OUT -m NS` requires NS, uses the loader's
recorded load order (a dep finishes loading before its requirer, so it's
topological) to concatenate the used namespaces into one .clj that runs on a
plain jolt with no deps/jpm. `jolt-deps uberscript` resolves first. The loader
records loaded file paths when ctx env has :loaded-files.
Phase 4 — conformance. deps-conformance-test resolves real pure-cljc git libs
and reports load/run status, gated behind JOLT_CONFORMANCE so CI stays offline.
First run: medley works; cuerdas hits a reader gap (filed); stuartsierra/
dependency uses Long/MAX_VALUE (JVM interop, out of scope).
Tests: uberscript-test, deps-conformance-test (skips without the flag).
Add --version/version, -e/--eval, -f/--file, -m/--main (require NS and apply
its -main to *command-line-args*), and nrepl-server/--nrepl-server taking an
optional [host:]port (nrepl stays an alias). repl/help/-h round it out.
Also rewrite doc/tools-deps.md as design notes for the now-implemented deps
support (it still read as pre-implementation research), and note the new flags
in the README. Test: cli-test runs the flags from source.
The deps tests built temp paths from $TMPDIR, which isn't set on the CI
runners — deps-resolve-test then tried to mkdir at the filesystem root and
failed. Default to /tmp.
A separate jolt-deps executable (mirroring jpm beside janet) resolves a
deps.edn into source roots and runs jolt with them on JOLT_PATH; the runtime
stays deps-agnostic.
- src/jolt/deps.janet: read deps.edn (Janet parses it directly), resolve
:git/* and :local/root via jpm/pm's download-bundle (git fetch + cache, no
build), recurse for transitive deps, return de-duped roots. jpm loaded
lazily so it's never embedded. Roots cached on a deps.edn hash.
- src/jolt/deps_cli.janet + project.janet: the jolt-deps tool — path/run/repl/-e.
- main: apply JOLT_PATH at runtime (the CLI ctx is frozen into the image at
build, so init's env read wouldn't see it).
Verified end to end against a local dep and a real git dep (medley). git only,
pure clj/cljc. Test: deps-resolve-test (local deps, no network).
The loader resolved a namespace against one hardcoded root (src/jolt, .clj
only). Now it searches an ordered list (ctx env :source-paths, stdlib first)
and tries .clj then .cljc. init adds roots from the :paths opt and JOLT_PATH.
This is the foundation for loading deps.edn libraries; verified loading medley
from an added root. No change to stdlib loading (3913 suite baseline held).
- core: pr-render/str-render now render a Jolt var as #'ns/name instead of
falling through to the generic table printer, which recursed into the var's
cyclic :meta/:ns refs and looped forever. Adds name-of/var-display helpers.
- jolt.nrepl: capture the post-eval namespace *after* running the form — jolt
evaluates map-literal values right-to-left, so {:val (eval form) :ns (the-ns)}
read the ns before in-ns ran, pinning every session to 'user'. Now in-ns/ns
switches persist across evals. render-value dropped: pr-str handles vars.
- specs: var printing (strings-spec) and nREPL in-ns/ns-persistence + explicit
:ns override (nrepl-test).
- host-interop-spec: add an 'interop / janet bridge' suite — janet/<name> and
janet.<module>/<name> resolution, the value-representation boundary (a Jolt
vector crosses as a Janet table), explicit-only (unprefixed module not
exposed), and unknown-symbol errors.
- nrepl-test: assert a def's value renders as #'ns/name (pr-str loops on a
var's cyclic ns refs, so jolt.nrepl renders vars itself).
Add a general Janet interop bridge and an nREPL implemented on top of it.
Interop bridge (evaluator):
- A qualified symbol whose namespace is `janet` or `janet.<module>` resolves
against Janet's environment: `janet/<name>` -> root binding (janet/slurp),
`janet.<module>/<name>` -> module binding (janet.net/server, janet.os/clock).
The explicit `janet` segment marks every crossing into host code (where
Clojure semantics, e.g. collection representation, no longer hold). This makes
the whole Janet stdlib — networking included — reachable from Clojure.
jolt.nrepl (Clojure, src/jolt/jolt/nrepl.clj):
- bencode codec (encode + streaming decode), ported from nrepl.bencode.
- server: accept loop via janet.net/accept + janet.ev/call (Janet's built-in
handler arity-checks Jolt closures, so we drive accept ourselves); ops clone/
describe/eval/load-file/close/ls-sessions/interrupt/eldoc following
babashka.nrepl response shapes. eval captures *out* by rebinding Janet's :out,
reports ns, streams out, and isolates the eval namespace (current-ns is global
ctx state) restoring it afterward. Vars are rendered as #'ns/name (pr-str
loops on a var's cyclic ns refs).
- client: connect / request / client-eval / client-clone / client-close.
CLI: `jolt nrepl [port]` starts the server and writes .nrepl-port; the Clojure
source is embedded at build time so the binary is self-contained from any cwd.
Tests: test/spec/nrepl-spec.janet (bencode), test/integration/nrepl-test.janet
(server+client over a real TCP/bencode wire, server in a subprocess).
Implement clojure.core futures backed by Janet's ev/thread for genuine
parallelism (CPU-bound work can use a second core, unlike cooperative go
blocks):
- future / future-call, deref + (deref f timeout-ms timeout-val), future?,
future-done?, future-cancel, future-cancelled?; realized? on futures.
- A worker OS thread computes and marshals back a [:ok v]/[:error e] result
over a thread-chan; a parent-side collector fiber caches it and closes a
broadcast latch so any number of deref-ers unpark.
- Snapshot semantics: separate heaps mean the body + captured state are copied
to the worker and only the result is copied back (mutating a captured atom
does not propagate). Documented in README.
- future-cancel can't interrupt a Janet OS thread, so it marks the future
cancelled/done (deref throws, predicates flip) while the worker runs out.
clojure-test-suite baseline 3915 -> 3913: implementing future unskips
realized_qmark.cljc's (when-var-exists future ...) block, which depends on
JVM Thread/sleep + real thread interruption jolt can't provide; deref then
re-raises the unresolved-Thread/sleep error. Documented at the baseline.
Spec: test/spec/futures-spec.janet (18 cases).
bootstrap-test.janet slurped a hardcoded /Users/yogthos/src/sci path (a
personal SCI clone, not vendor/sci) and had no assertions — a dev scratch
file that fails anywhere but the author's machine. SCI loading is already
covered by sci-bootstrap-test/sci-runtime-test against the vendored
submodule.
Two changes unlock native Janet speed in compile mode:
- Hot numeric primitives (+ - * < > <= >=) emit as native Janet SYMBOLS rather
than the variadic core fns, so Janet's compiler uses its arithmetic/compare
opcodes. = / not= / quot / rem / mod / division stay as core fns (their
semantics differ from Janet's). Trade-off: the strict non-number checks are
relaxed under compilation (documented perf-mode divergence).
- emit-invoke emits a DIRECT call (f arg...) when the callee is a function
reference (core/local/symbol/fn), instead of wrapping every call in jolt-call.
jolt-call is kept only for keyword/collection literals in call position
((:k m), ({:a 1} :a)) so IFn dispatch still works.
compiled fib(30): 3.4s -> 0.076s (native ceiling), faster than jank's 0.8s.
Updated compiler-test string assertions (core-+ -> +); compile-mode-test gains
native-op + IFn-dispatch cases; README documents compile mode. jpm test green.
Compile mode was broken: compiled defns interned only into the jolt namespace,
which Janet's eval couldn't see, so calling a compiled function threw 'unknown
symbol'. And load-string ignored :compile? entirely.
- Each context gets a persistent Janet env (ctx-janet-env), a child of the
compiler module env (so core-* resolve) holding the context's user defs.
compile-and-eval evals into it, so def/defn persist and resolve across forms;
contexts stay isolated. nil ctx (one-off eval) gets a fresh child.
- Emit a named fn for defn ((def f (fn f [..] (f ..)))) so recursion resolves
lexically (the anonymous form couldn't forward-reference f at compile time).
- Extract eval-one (per-form routing) and use it in both eval-string and
load-string, so load-string honors :compile?. Stateful forms still interpret.
compiled fib(30): ~50s (interpreted) -> 3.4s (~15x). spec: compile-mode-test
gains cross-form/recursion/def + context-isolation cases. jpm test green.
Fast operator emission (the rest of the win) is Phase 2.
