* core: Stage 2 Task 2 tier 2a — compile defprotocol/extend-type/extend-protocol
Applies the proven enabler: stateful primitives become per-ctx closures
captured over ctx, interned in clojure.core (install-stateful-fns!), so
they resolve + compile as plain :var invokes and work for deferred calls.
- protocol-dispatch / register-method: extracted from the interpreter
special handlers into ctx-taking impls (protocol-dispatch-impl /
register-method-impl) + interned as ctx-capturing clojure.core fns.
Removed their special-symbol? entries + handler arms, and dropped them
from host_iface special-names + compiler uncompilable-heads.
- defprotocol/extend-type/extend-protocol macros now pass the protocol/
method/type NAMES as strings (not symbols), so the emitted calls compile
as ordinary invokes; removed the three macros from special-names so the
analyzer expands+compiles them instead of punting to the interpreter.
- Both interpreter and compiled paths now call the same ctx-capturing
closures (one dispatch implementation, no special-form duplication).
reify/make-reified deferred to tier 2b (map-eval shape); defrecord waits
on deftype (tier 5).
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + spec (protocols 7/7, multimethods 9/9).
* core: Stage 2 Task 2 tier 2b — compile reify (make-reified as a fn)
Completes the protocol machinery: make-reified joins protocol-dispatch/
register-method as a ctx-capturing clojure.core fn (install-stateful-fns!).
- make-reified-impl takes the EVALUATED {keyword fn} method map (a phm when
compiled, struct/table when interpreted) and builds the reified object.
- reify macro passes the protocol NAME as a string; method map is an ordinary
map literal evaluating to {keyword fn}.
- Removed make-reified's special-symbol? entry + handler arm, and dropped
make-reified + reify from host_iface special-names + compiler
uncompilable-heads.
reify now compiles and dispatches in both modes (single- and multi-method).
With tier 2a, the full protocol surface (defprotocol/extend-type/
extend-protocol/reify) compiles; defrecord still waits on deftype (tier 5).
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + spec (protocols 7/7x3).
* core: Stage 2 Task 2 tier 3 — compile (var x) + binding
binding keys its thread-binding frame on var cells via (var x), so it
needed (var x) to compile. Added a the-var IR node that emits the embedded
var cell itself (vs var-ref, which derefs):
- ir.clj: the-var node.
- analyzer: 'var' added to handled; analyze-special resolves the symbol to
its var and emits the-var (uncompilable for a non-var, matching Clojure).
- backend: :the-var emits (quote cell) — the exact per-ctx cell var-get
keys on, so a compiled binding overrides + restores correctly.
- removed var from loader stateful-head? + host_iface special-names, and
binding from special-names so it expands+compiles.
Dynamic binding now compiles end-to-end (override/restore, and a compiled
fn reading the dynamic var under the binding) in both modes.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, clojure-test-suite >=4034/67, features 78/78,
all unit + spec (state/metadata).
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* core: compile macro expanders via staged bootstrap (Stage 2 Task 1)
Macros are now compiled, not interpreted, by steady state — matching
Clojure (macros are ordinary compiled fns the Java seed compiles for
clojure.core) and ClojureScript (macros compiled, invoked at compile
time). Neither reference keeps an interpreted-closure fallback.
The early macros (00-syntax) are defined while the self-hosted analyzer
is still being bootstrapped (it builds lazily after the overlay loads),
so macro-compile-hook returns nil and they get an interpreted closure.
The bootstrap compiler.janet can't substitute (it punts on syntax-quote,
which nearly every expander uses).
Fix = staged bootstrap, the same pattern as the compiler fixpoint:
defmacro stashes the expander source on the var (:macro-src) plus a
:macro-uses-env flag; once the overlay + analyzer are fully built,
backend/recompile-macros! (via ensure-macros-compiled! at the end of
load-core-overlay!) compiles each stashed expander through the now-live
analyzer and rebinds the var, marking :macro-compiled. Idempotent;
&env/&form macros keep the interpreted closure (the compiled fn* has no
such params). The interpreter is now a build-time crutch, gone by
steady state.
Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring
(which the analyzer punts on) to a plain rest param + (first rest), so
all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79.
47/47 overlay macros compiled, 0 interpreted; user macros compile
immediately post-init. Gate green: conformance 267x3, fallback-zero
31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap,
lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap,
features 78/78, all unit+spec, core-bench neutral.
* core: wrap macro expanders in fn (not fn*) so destructured arglists compile
Follow-up to the staged macro-compile change. The macro-recompile pass
wrapped each expander in the raw fn* primitive, which punts on a
destructuring rest param — so if-not/if-let/if-some/assert (using the
canonical '& [else]') couldn't compile and had been rewritten to plain
rest params as a workaround.
Root cause: fn* is the primitive; the fn MACRO is what desugars
destructuring (rest, map, nested) into the body before lowering. Wrapping
expanders in fn instead of fn* compiles any destructured macro arglist
uniformly, so the workaround is unnecessary — reverted those 4 macros to
the canonical '& [else]' forms.
Net: rest-destructuring is fully compiled for all normal code (fn/defn/
let/macro params). Only the hand-written raw fn* primitive still punts
(jolt-f79, downgraded to P4 — falls back to interpreter, still correct).
47/47 overlay macros compiled. Gate green: conformance 267x3,
fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host,
staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67,
sci-bootstrap, features, all unit+spec.
* core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79)
jolt-f79 asked to compile destructuring in fn* rest params. Checking
against Clojure inverts the premise: Clojure's fn* REJECTS destructuring
at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding
form, expected symbol'). So the self-hosted analyzer was already correct
— fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn
MACROS desugar destructuring. The real defect was the interpreter
leniently destructuring raw fn*, and defn emitting raw fn* to rely on it.
Changes:
- evaluator: fn*/let*/loop* now reject non-symbol binding forms with
Clojure's exact messages (require-symbol-params/plain-sym?), so the
interpreter agrees with the analyzer + Clojure.
- 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring
params desugar; unnamed, so self-recursion still resolves via the var.
- 00-syntax: completing that exposed a real gap — the overlay destructure
fn didn't handle kwargs (a map pattern bound against a fn's sequential
rest); it had only worked via the interpreter's destructure-bind. Added
the seq->map coercion to the map? branch (sequential: 1 map elt => that
map, else apply hash-map), matching destructure-bind so interpret ==
compile.
Net: fn*/let*/loop* are plain-symbol primitives across interpreter,
analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro
params, incl kwargs & {:keys}) compiles through the macros with no
interpreter fallback. Regression spec added.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44,
clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all
unit+spec (destructuring 50/50).
* docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79)
grammar.ebnf: the destructuring note already attributed patterns to the
binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES
they desugar to take plain symbols only (a non-symbol binding errors, as
in Clojure).
self-hosting-compiler.md: the 'compile destructuring via a shared
destructure expander instead of falling back' item is done (and its
jolt-7dl ref was stale) — destructuring now compiles through the
fn/let/loop/defn macros' desugaring; the primitives reject patterns.
* core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy
First slice of moving stateful forms onto the compile path (jolt-eaa).
- loader stateful-head?: drop syntax-quote (the analyzer's `handled` set
already compiles it; routing it to the interpreter was redundant).
- host_iface special-names: drop definterface/extend/proxy (stub macros
expanding to def/nil — their expansions compile once unpunted).
- letfn stays interpreted: its let* expansion needs letrec semantics
(mutual recursion between the fns), which sequential compiled let* lacks
— a later tier.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + protocol/multimethod/macro specs.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
* compiler: self-hosted analyzer compiles set literals (#{…})
Stage 1 Task 1. analyzer.clj punted set literals to the interpreter
((form-set? form) (uncompilable "set literal")); now it builds the set-node IR
(already defined in ir.clj) from (form-set-items form), and backend.janet emits
(make-phs e1 e2 …) — each element evaluated then the persistent set built,
mirroring compiler.janet's emit-set-expr and the interpreter's :jolt/set path.
Closes a self-hosted-analyzer vs bootstrap-compiler parity gap: #{…} no longer
forces interpreter fallback on the compile path.
Gate: conformance 262x3 (+4 set-literal cases incl computed elements / empty /
in-let), fixpoint, self-host, sci, suite 3981/66, specs+unit green; core-bench
neutral (A/B). set?/disj-as-fns remain deliberately interpreted (in-sync across
all three lists) — adjudicated in Task 2.
* test: fallback-zero harness — assert non-stateful forms compile (not interpret)
Stage 1 Task 3. self-host-test checks results but not which path ran. This runs
the portable analyzer (backend/analyze-form) on a corpus of non-stateful forms
and asserts NONE raise :jolt/uncompilable — i.e. the self-hosted analyzer
compiled them, not the interpreter fallback. Inverse sanity list confirms a few
intentional-interpret forms (ns/defmacro/require/set?/letfn) still punt, so the
harness can't pass by compiling everything.
29 must-compile (incl set literals from Task 1) + 5 must-punt, 0 failures. As
Stage 1 parity grows, forms move from the punt list into must-compile; when the
fallback set equals the frozen intentional stateful set, the bootstrap is
retireable.
* core: migrate 7 lazy seq fns from the Janet seed to the Clojure overlay (40-lazy)
Finishes a port the prior team started and reverted (bb4a3e0): the 40-lazy.clj
tier moved lazy seq fns Janet→Clojure but regressed the suite to 849 because
lazy-seq's expansion leaked as data in compile mode — that was jolt-r81, since
root-fixed (lazy-seq/lazy-cat moved to 00-syntax). With the wall gone, the port
works. This shrinks the Janet seed toward the north star (self-hosted
clojure-in-clojure on a minimal host bootstrap).
Moved to core/40-lazy.clj (wired as a loaded tier after 30-macros):
distinct keep keep-indexed map-indexed cycle repeat iterate
40-lazy.clj completed to full parity: distinct gains its transducer arity;
keep/keep-indexed/map-indexed already had both arities.
Removed from the Janet seed (core.janet): the 7 core-* fns + their core-bindings
entries, the now-dead td-keep/td-map-indexed transducer helpers (the CLJ versions
carry their own), and the already-dead core-partition-by/core-xml-seq (shadowed by
10-seq/20-coll). Net: core.janet −131 lines.
Deferred (kept in Janet, separate follow-ups): partition-all (a CLJ port via
take/drop realizes a non-minimal element count, tripping the §6.3 laziness
counters + a suite file) and repeatedly (canonical CLJ doesn't validate args, so
the repeatedly.cljc throw cases regress). Both need behavior-matching first.
Gate: conformance 262x3, lazy-infinite 44/44, clojure-test-suite 4004/66 (UP from
3981 — the CLJ versions add coverage, e.g. distinct value-equality), fixpoint,
self-host, sci 422/0, fallback-zero, specs+unit, core-bench all green.
* test: raise clojure-test-suite baseline 3981 -> 4004 (lazy-fn migration coverage)
* core: migrate partition-all to the Clojure overlay (minimal realization)
Resolves the deferred partition-all port (jolt-yo3). The earlier CLJ attempt via
lazy take/drop over-realized vs the Janet pstep, tripping the §6.3 laziness
counter. The collection arities now realize EXACTLY n per chunk with a first/rest
loop and continue from the advanced cursor (no re-drop), so (take 3 (partition-all
2 (map counting (range)))) realizes exactly 6 — matching minimal realization in
both interpret and compile modes. Keeps transducer + [n coll] + [n step coll]
arities. letfn-bound recursion sidesteps the compile-mode multi-arity closure bug
(jolt-zxw), like keep-indexed/map-indexed.
Removed from the Janet seed: core-partition-all + its binding + the now-dead
td-partition-all helper (the CLJ version carries its own transducer arity).
Gate: conformance 262x3, lazy-infinite 44/44 (incl the §6.3 partition-all
counter), clojure-test-suite 4004/66, fixpoint, self-host, specs+unit green.
* core: migrate repeatedly to Clojure + fix char-not-callable / take count validation
Resolves the deferred repeatedly port (jolt-8qx). The blockers were two jolt
leniencies vs Clojure, now fixed (and correct beyond repeatedly):
- A char (a :jolt/type-tagged struct) fell into the struct-as-map branch of both
jolt-call (compile path) and the interpreter's apply dispatch, so (\a) returned
nil instead of throwing. Now only an UNtagged struct (a map literal) — or a
record — is callable as a key lookup; tagged structs fall through to "Cannot
call … as a function". Symbols are still handled (keyword-style get).
- core-take didn't validate its count, letting Janet's >= silently compare an int
to a char/string. It now rejects a non-number n like Clojure.
With those, the canonical CLJ repeatedly matches: (first (repeatedly non-fn)) and
(repeatedly non-number f) throw. Moved repeatedly to core/40-lazy.clj; removed
core-repeatedly + its binding from the seed.
These correctness fixes help broadly: repeatedly.cljc goes clean (19/10 -> 29/0),
and the suite rises 4004 -> 4034 pass / 66 -> 67 clean. Baseline raised.
Gate: conformance 262x3, lazy-infinite 44/44, clojure-test-suite 4034/67,
fixpoint, self-host, sci, fallback-zero, specs+unit green.
* test: document the 4004 -> 4034 baseline raise (partition-all/repeatedly + char/take fixes)
* docs: partition-all letfn is for minimal realization, not jolt-zxw
jolt-zxw (multi-arity arity-param mis-capture in a nested lazy-seq under :compile?)
is no longer reproducible: an arity-direct partition-all now compiles correctly in
the overlay. The original failure was an artifact of the CLJ multi-arity version
coexisting with the Janet core-partition-all ([n & rest]) during migration — the
shadowing confused multi-arity dispatch; removing the Janet version resolved it.
The letfn in partition-all stays purely for minimal realization (jolt-yo3).
* compiler: compile set?/disj as plain fns (close the last Stage-1 fallback gap)
Stage 1 jolt-g3h. set? and disj were special-cased in all three "can't compile"
lists (host_iface special-names, compiler.janet uncompilable-heads, evaluator
special-symbol? + handlers) — but they're pure value-production with callable
core vars (core-set?/core-disj), and those vars are byte-for-byte equivalent to
the evaluator handlers. Removed them from all three lists + dropped the now-dead
evaluator handler arms, so they're ordinary clojure.core fns everywhere: the
analyzer compiles (set? x)/(disj s x) as normal var calls instead of punting to
the interpreter.
Verified identical results in default AND JOLT_MUTABLE builds (no representation
sensitivity — sets are phs in both, unlike vector?/list? which collapse).
With this, the self-hosted analyzer's compile-path fallback set equals the frozen
intentional stateful set (Task 2) — it's now a strict superset of the bootstrap
compiler's compilable surface, so the Janet bootstrap is retireable (Stage 2).
fallback-zero: set?/disj moved to must-compile (31 now), set! into must-punt.
Gate: conformance 267x3 (+5 set?/disj cases), lazy-infinite 44/44, suite 4034/67,
fixpoint, self-host, sci, specs+unit green.
adds self-hosted compiler is functionally:
- The default compile path is the portable pipeline using jolt.analyzer (Clojure) → host-neutral IR → backend.janet.
- The analyzer is itself Clojure, compiled by jolt for true self-hosting.
- bootstrap-fixpoint passes (stage1 == stage2 == stage3): rebuilding the compiler on its own output.
- clojure.core is now self-hosted in the overlay.
- Stateful forms (defmacro/ns/deftype/defmulti/require/in-ns) are interpreted by design.
Destructuring (drove by trying to load Selmer):
- defmacro params now destructure like fn (parse-params + destructure-bind), so
[& [a & more :as all]] and {:keys ...} work in macro arglists.
- map-destructuring a sequential value treats it as keyword args (alternating
k/v, or a trailing map) — the [& {:keys [...]}] form, in fns and macros.
- :keys/:syms accept namespaced symbols (x/y): looks up the namespaced key,
binds the bare local.
deps.edn: read with Jolt's reader instead of Janet's parse, so EDN ';' line
comments are handled (Janet treats ';' as splice).
clojure.java.io: the shim was at clojure/java_io.clj (ns clojure.java-io, never
the clojure.java.io that code requires) and used bare-qualified Janet calls that
the janet.* bridge no longer resolves. Moved to clojure/java/io.clj and rewritten
on janet.file/os: file/reader/writer/resource/copy/delete-file/make-parents.
Specs in destructuring-spec; destructuring note in the ebnf.
The reader expanded ^X form to (with-meta form X), which evaluated the tag (so
^String errored 'Unable to resolve symbol: String') and, as a param, was no
longer a bare symbol (so the arg bound to nil). Now a keyword/symbol/string hint
on a symbol attaches to the symbol's :meta and the symbol stays bare, so type
hints are transparent in params, lets, and bodies. Map metadata still uses a
runtime with-meta form.
meta now returns a symbol's :meta, and def applies the name's metadata
(^:dynamic, ^:private, ^Type tag, ^{:doc}) to the var, so (meta (var x)) is
consistent. Specs in metadata-spec; grammar note in the ebnf.
README notes regex \p{...} as unsupported (separate from this).
read-char read the char name with symbol-char?, so a literal whose char isn't a
symbol char (\{, \(, \), \,, \%, …) came back as an empty name and errored
'Unsupported character'. Now a single non-symbol char after \ is taken as a
one-character literal of that char. Surfaced by funcool/cuerdas (uses \{), which
now loads. Spec cases added.
The Clojure stdlib namespaces (clojure.string/set/walk/edn/zip, jolt.http/
interop/shell/nrepl) were loaded cwd-relative, so the built binary couldn't
load them outside the repo. Now stdlib_embed slurps them all at build time into
a {ns -> source} map frozen in the image, and the loader falls back to it when a
namespace isn't found on disk. The artifact is self-contained — clojure.string
works from any directory.
Drops the one-off nREPL source embed (jolt.nrepl is covered by this now). The
baked-in stdlib is excluded from uberscript bundles since it's part of the
runtime. clojure.core is unchanged (Janet, auto-referred).
Test: embedded-stdlib-test requires from the embedded copy with FS roots cleared.
Phase 3 — bundling. `jolt uberscript OUT -m NS` requires NS, uses the loader's
recorded load order (a dep finishes loading before its requirer, so it's
topological) to concatenate the used namespaces into one .clj that runs on a
plain jolt with no deps/jpm. `jolt-deps uberscript` resolves first. The loader
records loaded file paths when ctx env has :loaded-files.
Phase 4 — conformance. deps-conformance-test resolves real pure-cljc git libs
and reports load/run status, gated behind JOLT_CONFORMANCE so CI stays offline.
First run: medley works; cuerdas hits a reader gap (filed); stuartsierra/
dependency uses Long/MAX_VALUE (JVM interop, out of scope).
Tests: uberscript-test, deps-conformance-test (skips without the flag).
Clean up the .nrepl-port file when the server stops — via defer on a clean
unwind and SIGINT/SIGTERM handlers for Ctrl-C. A SIGKILL still can't be caught,
so it may occasionally be left behind.
Add --version/version, -e/--eval, -f/--file, -m/--main (require NS and apply
its -main to *command-line-args*), and nrepl-server/--nrepl-server taking an
optional [host:]port (nrepl stays an alias). repl/help/-h round it out.
Also rewrite doc/tools-deps.md as design notes for the now-implemented deps
support (it still read as pre-implementation research), and note the new flags
in the README. Test: cli-test runs the flags from source.
A separate jolt-deps executable (mirroring jpm beside janet) resolves a
deps.edn into source roots and runs jolt with them on JOLT_PATH; the runtime
stays deps-agnostic.
- src/jolt/deps.janet: read deps.edn (Janet parses it directly), resolve
:git/* and :local/root via jpm/pm's download-bundle (git fetch + cache, no
build), recurse for transitive deps, return de-duped roots. jpm loaded
lazily so it's never embedded. Roots cached on a deps.edn hash.
- src/jolt/deps_cli.janet + project.janet: the jolt-deps tool — path/run/repl/-e.
- main: apply JOLT_PATH at runtime (the CLI ctx is frozen into the image at
build, so init's env read wouldn't see it).
Verified end to end against a local dep and a real git dep (medley). git only,
pure clj/cljc. Test: deps-resolve-test (local deps, no network).
The loader resolved a namespace against one hardcoded root (src/jolt, .clj
only). Now it searches an ordered list (ctx env :source-paths, stdlib first)
and tries .clj then .cljc. init adds roots from the :paths opt and JOLT_PATH.
This is the foundation for loading deps.edn libraries; verified loading medley
from an added root. No change to stdlib loading (3913 suite baseline held).
- core: pr-render/str-render now render a Jolt var as #'ns/name instead of
falling through to the generic table printer, which recursed into the var's
cyclic :meta/:ns refs and looped forever. Adds name-of/var-display helpers.
- jolt.nrepl: capture the post-eval namespace *after* running the form — jolt
evaluates map-literal values right-to-left, so {:val (eval form) :ns (the-ns)}
read the ns before in-ns ran, pinning every session to 'user'. Now in-ns/ns
switches persist across evals. render-value dropped: pr-str handles vars.
- specs: var printing (strings-spec) and nREPL in-ns/ns-persistence + explicit
:ns override (nrepl-test).
Add a general Janet interop bridge and an nREPL implemented on top of it.
Interop bridge (evaluator):
- A qualified symbol whose namespace is `janet` or `janet.<module>` resolves
against Janet's environment: `janet/<name>` -> root binding (janet/slurp),
`janet.<module>/<name>` -> module binding (janet.net/server, janet.os/clock).
The explicit `janet` segment marks every crossing into host code (where
Clojure semantics, e.g. collection representation, no longer hold). This makes
the whole Janet stdlib — networking included — reachable from Clojure.
jolt.nrepl (Clojure, src/jolt/jolt/nrepl.clj):
- bencode codec (encode + streaming decode), ported from nrepl.bencode.
- server: accept loop via janet.net/accept + janet.ev/call (Janet's built-in
handler arity-checks Jolt closures, so we drive accept ourselves); ops clone/
describe/eval/load-file/close/ls-sessions/interrupt/eldoc following
babashka.nrepl response shapes. eval captures *out* by rebinding Janet's :out,
reports ns, streams out, and isolates the eval namespace (current-ns is global
ctx state) restoring it afterward. Vars are rendered as #'ns/name (pr-str
loops on a var's cyclic ns refs).
- client: connect / request / client-eval / client-clone / client-close.
CLI: `jolt nrepl [port]` starts the server and writes .nrepl-port; the Clojure
source is embedded at build time so the binary is self-contained from any cwd.
Tests: test/spec/nrepl-spec.janet (bencode), test/integration/nrepl-test.janet
(server+client over a real TCP/bencode wire, server in a subprocess).
Implement clojure.core futures backed by Janet's ev/thread for genuine
parallelism (CPU-bound work can use a second core, unlike cooperative go
blocks):
- future / future-call, deref + (deref f timeout-ms timeout-val), future?,
future-done?, future-cancel, future-cancelled?; realized? on futures.
- A worker OS thread computes and marshals back a [:ok v]/[:error e] result
over a thread-chan; a parent-side collector fiber caches it and closes a
broadcast latch so any number of deref-ers unpark.
- Snapshot semantics: separate heaps mean the body + captured state are copied
to the worker and only the result is copied back (mutating a captured atom
does not propagate). Documented in README.
- future-cancel can't interrupt a Janet OS thread, so it marks the future
cancelled/done (deref throws, predicates flip) while the worker runs out.
clojure-test-suite baseline 3915 -> 3913: implementing future unskips
realized_qmark.cljc's (when-var-exists future ...) block, which depends on
JVM Thread/sleep + real thread interruption jolt can't provide; deref then
re-raises the unresolved-Thread/sleep error. Documented at the baseline.
Spec: test/spec/futures-spec.janet (18 cases).
Two changes unlock native Janet speed in compile mode:
- Hot numeric primitives (+ - * < > <= >=) emit as native Janet SYMBOLS rather
than the variadic core fns, so Janet's compiler uses its arithmetic/compare
opcodes. = / not= / quot / rem / mod / division stay as core fns (their
semantics differ from Janet's). Trade-off: the strict non-number checks are
relaxed under compilation (documented perf-mode divergence).
- emit-invoke emits a DIRECT call (f arg...) when the callee is a function
reference (core/local/symbol/fn), instead of wrapping every call in jolt-call.
jolt-call is kept only for keyword/collection literals in call position
((:k m), ({:a 1} :a)) so IFn dispatch still works.
compiled fib(30): 3.4s -> 0.076s (native ceiling), faster than jank's 0.8s.
Updated compiler-test string assertions (core-+ -> +); compile-mode-test gains
native-op + IFn-dispatch cases; README documents compile mode. jpm test green.
Compile mode was broken: compiled defns interned only into the jolt namespace,
which Janet's eval couldn't see, so calling a compiled function threw 'unknown
symbol'. And load-string ignored :compile? entirely.
- Each context gets a persistent Janet env (ctx-janet-env), a child of the
compiler module env (so core-* resolve) holding the context's user defs.
compile-and-eval evals into it, so def/defn persist and resolve across forms;
contexts stay isolated. nil ctx (one-off eval) gets a fresh child.
- Emit a named fn for defn ((def f (fn f [..] (f ..)))) so recursion resolves
lexically (the anonymous form couldn't forward-reference f at compile time).
- Extract eval-one (per-form routing) and use it in both eval-string and
load-string, so load-string honors :compile?. Stateful forms still interpret.
compiled fib(30): ~50s (interpreted) -> 3.4s (~15x). spec: compile-mode-test
gains cross-form/recursion/def + context-isolation cases. jpm test green.
Fast operator emission (the rest of the win) is Phase 2.
(chan n xform) applies a transducer on the put side: a jolt transducer is a
directly-callable closure, composed over a reducing fn whose step gives each
output value into the channel (honoring its buffer kind). One put may yield zero
or more values; a reduced result (e.g. from take) closes the channel; close!
runs the transducer completion arity to flush stateful remainders. Works with
map/filter/mapcat/take/comp/etc.
Buffers: (buffer n) fixed, (dropping-buffer n) drops new values when full,
(sliding-buffer n) drops the oldest. Implemented via a non-blocking give —
(ev/select [ch v] closed-chan) detects a full buffer without parking.
harness: run-spec flushes per suite. spec: core.async/channel-transducers (5),
core.async/buffers (3). jpm test green.
Note: distinct/dedupe/partition-all/partition-by still lack a 0-coll transducer
arity in core (separate gap), so they can't yet be used as channel xforms.
Jolt's dynamic-var binding stack was a single global array, so concurrent go
blocks interleaved each other's bindings and a go block didn't see the bindings
in effect when it was spawned.
Move the binding stack into Janet's fiber-local dyn (:jolt/binding-stack): each
fiber (go block) lazily gets its own array, so bindings can't interleave. Janet
ev/go fibers inherit the parent's dyn, but go-spawn now snapshots the binding
stack at spawn time and installs a private copy in the new fiber — Clojure
binding conveyance. A go block's own (binding ...) shadows the conveyed frame.
types.janet: cur-binding-stack / snapshot-bindings / install-bindings; var-get/
var-set/push/pop use the fiber-local stack. async.janet: go-spawn conveys.
spec: core.async/binding-conveyance (4 cases — conveyance, isolation, no leak to
root, inner shadowing). jpm test green.
Janet fibers are stackful coroutines, so a go block is just its body run in a
fiber that parks on channel ops by yielding to the event loop — the interpreter
call stack rides along, no CPS/state-machine transform. So <!/>! work anywhere
(inside try, nested fns, loops), unlike Clojure's go macro.
src/jolt/async.janet implements chan/chan?/close!/<!/>!/<!!/>!!/go/go-loop/
thread/alts!/timeout/put!/take! over ev/ channels and fibers, installed as the
clojure.core.async namespace (pre-populated in init, so require finds it).
A channel is a pair of ev/chans (:ch values + :done close-signal); a take is
(ev/select :ch :done), which drains buffered values before the close signal —
giving Clojure's drain-then-nil semantics without the buffer loss of
ev/chan-close, and with no leaked fibers (close! just closes :done).
Single-threaded cooperative scheduling: <! (park) and <!! (block) coincide.
Dynamic-var conveyance (Phase 2) and channel transducers (Phase 3) are TODO.
spec: test/spec/core-async-spec (16 cases — go/channels, buffering+close,
go-loop pipelines, alts!/timeout, parking inside try/nested-fn). jpm test green.
- A map entry is a 2-element tuple (Jolt produces tuples only from map iteration;
vector literals are pvecs, lists are arrays). key/val/map-entry? now accept a
2-tuple and reject a plain vector, matching Clojure's MapEntry-vs-vector
distinction — no metadata needed, the representations already differ.
- min-key/max-key reproduce Clojure's NaN-aware folding (2-arg strict </>, then
<=/>=) and require numeric keys (NaN allowed, strings throw).
- subvec coerces float/NaN indices like (int ...) (truncate, NaN->0) then
bounds-checks, instead of throwing on non-integers.
min_key 35/14 -> 49/0 (clean); key/val recover the 2-vector cases; subvec floats
fixed. clojure-test-suite pass 3898->3921. Updated conformance-test (key/val now
needs a real entry). spec: map/map-entry-&-key-ordering (14). jpm test green.
- subs requires a string and validates 0<=start<=end<=count (no Janet
from-end/clamping); negative/out-of-range/nil indices throw
- assoc! on a transient vector bounds-checks the index (0..count)
subs 11-fail -> 24/5 (5 remaining are byte-vs-codepoint Unicode, platform);
assoc_bang 32/6 -> 35/3. clojure-test-suite pass 3889->3898.
spec: string/subs-strictness (7), transient/assoc!-bounds (4). jpm test green.
merge now throws when a non-first arg is a scalar, a set, a list, or a
wrong-length vector (a length-2 vector or a map still merge). Other map-like
tables (records/sorted-maps/host tables, e.g. SCI's namespaces) keep the lenient
conj path so the SCI bootstrap still loads.
merge 29/11/2 -> 35/3/4. clojure-test-suite pass 3874->3880.
spec: 5 merge strictness cases. jpm test green.
- first throws on scalars (numbers/keywords/booleans/char & symbol structs)
- rseq is vector/sorted-only (throws on strings/maps/numbers/seqs)
- assoc requires an even kv count and a map/vector/nil receiver
first clean; rseq 12/1; assoc 39/3. clojure-test-suite pass 3864->3874.
spec: seq/more-strictness (11). jpm test green.
- assoc on a vector bounds-checks the index (0..count); out-of-range/negative throw
- dissoc throws on non-maps (numbers/sequences/sets/scalars); nil ok; records/
sorted-maps/meta-maps still handled
- count throws on scalars (numbers/keywords/symbols/booleans/chars)
- subvec validates vector type and 0<=start<=end<=count
- numerator/denominator always throw (Jolt has no ratio type)
- min-key/max-key throw on no values
count 18/2; dissoc 19/3; assoc 36/6; subvec 26/3/5; numerator/denominator throw
cases pass. clojure-test-suite pass 3840->3864. spec: map/strictness (16). jpm test green.
- peek/pop are stack-only (vectors/lists): throw on sets/maps/strings/scalars,
and pop throws on an empty vector/list
- vec throws on non-seqable args (numbers/keywords/transients)
- key/val require a map entry (2-element vector); throw on nil/numbers/maps/sets
pop clean; peek 9/2; vec 17/2/1; key/val 12/2/1 (remaining are the
2-vector-vs-MapEntry / tuple-from-seq divergences). pass 3824->3840.
spec: seq/accessor-strictness (16). jpm test green.
conj!/assoc!/dissoc!/disj!/pop!/persistent! now throw on a non-transient (or
wrong transient kind) instead of falling back to the persistent op, matching
Clojure. conj! keeps its special arities: (conj!) -> (transient []), (conj! coll)
-> coll. conj! onto a transient map accepts a [k v] pair or a map (merge), and
throws on a list/set/seq.
pop_bang/dissoc_bang clean; conj_bang 13/1/22->47/3/1; persistent_bang 9/8->16/1.
clojure-test-suite pass 3781->3824. spec: transient/strictness (10). jpm test green.
- zero?/pos?/neg? throw on non-numbers; odd?/even? throw on non-integers
(nil, infinities, NaN, fractional) via need-num/need-int helpers
- comparisons < > <= >= throw on non-number args (1-arity stays true, no check)
- max/min throw on non-number args
- quot/rem/mod throw on zero divisor and non-finite operands
odd?/even?/lt/gt/lt_eq/gt_eq suite files now clean; pass 3691->3738.
Updated systematic-coverage-test (zero? nil now throws). spec:
numbers/strictness (15). jpm test green.
Numbers printed via Janet's (string v) rendered infinities/NaN as inf/-inf/nan.
Add fmt-number so str/pr-str (and collection rendering) emit Infinity/-Infinity/
NaN. str.cljc 3->32 pass. Remaining str fails are the integer-valued-double
divergence ((str 0.0) is "0" not "0.0" since 0.0 == 0 in Janet).
spec: numbers/printing-of-inf-&-nan (5). jpm test green.
- case: quote list literals (read as arrays) in constant position so a wrapped
list ((a b c)) matches by value instead of being evaluated as a call; symbol
constants already quoted. Vector/map/set constants already worked. case errors
in the suite drop to 0 (60 pass).
- associative?: true only for vectors (pvec) and maps (phm/struct/sorted-map),
not lists/tuples-from-seq-fns/lazy-seqs/sets.
- reversible?: true for vectors and sorted-map/sorted-set only.
- update: coerce f via as-fn so (update m k :kw)/(update m k a-set) work; extra
args already handled.
- nth: (nth nil i)/(nth nil i default) returns nil/default instead of throwing.
clojure-test-suite pass 3649->3678, errors 122->105, clean files 44->46.
associative?/reversible? files now fully clean. spec: predicates + control/case.
jpm test green.
Biggest fix: jolt keywords are Janet keywords and maps are Janet structs/phm, but
(:a struct) at the Janet level returns nil (not a Clojure accessor) and errors on
a phm — so core-map/filter/sort-by/group-by/etc. calling (f x) directly broke the
ubiquitous keyword/set/map-as-function idioms ((map :a coll), (sort-by :k coll),
(filter a-set coll), (group-by :type coll)). Added as-fn coercion (keyword/symbol
-> key lookup, map -> key lookup, set -> membership) applied at the entry of
map/filter/remove/keep/mapv/filterv/sort-by/group-by/partition-by/some/
not-any?/not-every?/take-while/drop-while/min-key/max-key.
Also:
- realize-for-iteration treats nil as an empty seq (Clojure semantics), fixing
nth/nthrest/nthnext/take-last/reduce/doseq over nil.
- take-nth and interpose gained their 1-arg transducer arities.
- sort-by gained the 3-arg (keyfn comparator coll) form.
- min-key/max-key: single item returns it without calling f; ties keep the last.
- underive gained the 2-arg global-hierarchy form.
clojure-test-suite pass 3535->3649, errors 177->122, clean files 39->44.
spec: seq/IFn-values-as-functions (11). jpm test green.
Reader gaps caused the clojure-test-suite worker to crash whole deftests on
literals it could not parse (0N, 1.5M, 2r1010, 1/2), losing every assertion in
the file. read-number now handles:
- N (bigint) / M (bigdec) suffixes -> plain number (Jolt has no bignum/bigdec)
- ratios a/b -> double quotient
- radix integers NrDDD (2r1010, 16rFF, 36rZ) parsed by base
- exponents (1e3, 1.5e-2) and 0X hex
Also fixed suite measurement: when-var-exists now skips silently (its SKIP
print to stdout was corrupting the worker's count line, dropping whole files),
and the worker emits counts on an @@COUNTS sentinel line (robust against test
bodies that print, e.g. with-out-str). Runner parses the sentinel; deftest
crashes now report the underlying message.
Impact: clojure-test-suite 210->231 files run, pass 1955->3535, clean files
24->39. Baseline raised to 3450/38.
spec: numbers/literal-syntax (13 cases). jpm test green.
The reader now reads the symbolic float values ##Inf, ##-Inf and ##NaN. Added
infinite? and NaN? predicates. Fixed intval? to exclude infinity (floor(inf)=inf
but inf isn't integer-valued), so float?/double? are true for ##Inf and
int?/pos-int?/nat-int?/neg-int? are false for it.
This unblocked many number-test files whose forms previously failed to
READ (##Inf/##NaN literals), so clojure-test-suite jumped from 2241 to 2539
assertions and pass 1719 -> 1955. Baseline raised to 1900. NaN_qmark now runs.
float?/double? on integer-valued doubles (1.0) remain false: Janet represents
an integer and an integer-valued double identically, so they're inherently
indistinguishable — documented in the README Numbers section.
spec: numbers/floats-&-symbolic-values (15 cases). jpm test green.
Closes jolt-fy8 (fixable parts; int-vs-float ambiguity is a documented divergence).
Real transient correctness gaps surfaced by the clojure-test-suite:
- Transients are now invokable for read-only lookup like their persistent forms:
((transient v) i), ((transient m) k [default]), (:k (transient m)),
((transient s) x). jolt-invoke/coll-lookup gained a transient branch
(transient-lookup) that indexes :arr / canon-keyed :tbl. Added phm/canon as a
public canonicalizer so collection keys compare by value here too.
- assoc! accepts an ODD arg count (a missing final value is nil), unlike assoc —
core-assoc! now uses nil-safe get for the value instead of erroring.
- Using a transient after persistent! (or a second persistent!, or pop! on an
empty transient vector) now throws, via a :jolt/persistent invalidation flag
checked by the mutating ops. Catches the classic transient footgun.
spec: transients-spec gains invokable-lookup (7), assoc!-odd-args (4),
invalidation (4). clojure-test-suite pass 1704->1719.
Remaining transient fails are accepted lenient divergences (jolt doesn't throw
on bad-shape conj!/assoc!/pop! of wrong-typed args; nil set elements). jpm test green.
transduce-reduce and core-reduce both called realize-for-iteration, fully
realizing the coll before the reduce loop — so an infinite lazy seq hung before
the reduced short-circuit could fire. (into [] (take 5) (range)) etc. never
terminated.
Add reduce-with-reduced, which steps a lazy seq one cell at a time
(realize-ls/ls cell protocol), checking reduced? after each element so a take/
take-while transducer (or any reducing fn returning reduced) terminates over an
infinite seq. Route transduce-reduce and both core-reduce arities through it;
2-arg reduce now seeds from the first element and reduces the rest lazily.
spec: transducers/short-circuit + reduce/honors-reduced (13 cases).
clojure-test-suite timeouts 7->6, pass 1683->1688. jpm test green.
Fixes jolt-kxb.
Run the external cross-dialect clojure-test-suite (lread/clojure-test-suite,
240 per-fn .cljc files in clojure.test format) against Jolt:
- test/support/clojure_test.clj: minimal clojure.test + portability shims
(deftest/is/testing/are + when-var-exists/thrown?/big-int?/lazy-seq?) — just
enough surface to load the suite and tally pass/fail/error. Pre-loaded so the
suite's (require [clojure.test ...]) finds it already populated.
- test/integration/suite-worker.janet: one-shot worker that loads the shim,
the suite's number-range helper ns, and a single .cljc file, then prints
'pass fail error'.
- test/integration/clojure-test-suite-test.janet: spawns a worker per file
under an ev/with-deadline wall-clock budget (so infinite-seq tests that hang
Jolt's eager evaluator are auto-contained, not a manual skip-list) and asserts
pass/clean-file counts stay at/above a baseline. References ~/src/clojure-test-suite
if present; skips cleanly when absent, like the jank battery.
Current: 210 files run, 7 timed out, 2233 assertions -> 1683 pass / 350 fail /
200 error, 23 clean files. Remaining fails are genuine divergences (float/ratio/
bigint, lenient transients where Clojure throws), tracked separately.
Fixes two real evaluator bugs the suite surfaced:
- :refer now preserves a referred macro's :macro flag (was interned as a plain
value, degrading referred macros to functions).
- resolve-var now resolves ns aliases (like resolve-sym), so aliased macros
(e.g. p/thrown? via :as p) dispatch as macros instead of being called as fns.
Replace the correctness-only transient aliases with real mutable scratch
collections via host interop:
- transient vector -> a Janet array; conj!/assoc!/pop! mutate in place
- transient map -> a Janet table keyed by canonical key (collection keys still
compare by value); assoc!/dissoc!/conj! mutate in place
- transient set -> a Janet table; conj!/disj! mutate in place
- persistent! freezes back to a pvec / phm / phs
- count/nth/get/contains? work on transients; transient? predicate added
Building a map/set this way avoids the persistent path's per-step bucket-array
copying (transient map build ~35% faster at 20k here); vectors are comparable
since pvec conj is already ~O(1). The mutating ops return the transient and the
source collection is untouched.
spec/transients-spec (34 cases). conformance 218/218, jpm test green.
Close jolt-dd5.
- catch now binds the originally-thrown value (unwrapping the :jolt/exception
envelope), so (catch ... e (throw e)) rethrows the same exception instead of
nesting another envelope, and (catch ... e e) on (throw 42) yields 42.
- var-set updates the innermost thread-binding frame for the var (replacing the
stack slot) when the var is dynamically bound, matching Clojure; it falls back
to the root otherwise.
Restored spec cases: exceptions rethrow + catch-binds-thrown-value, namespaces
var-set-in-binding. conformance 218/218, jpm test green.
Close jolt-do7. Maps/sets keyed by a collection (a map, vector, ...) now compare
by value instead of Janet identity:
- PHM/PHS hash and compare keys through an injected canonicalizer (collection
keys -> value-hashable struct/tuple); keys are still stored as-is
- map literals and core-assoc promote to a phm when a key is a collection
- frequencies/group-by use a phm base so collection elements/keys dedup by value
- set equality is value-based (from earlier)
Real bugs found and fixed along the way:
- set literals #{(inc 1)} did not evaluate their elements (stored raw forms!)
- the REPL printer rendered phm maps as {} (they hit the deftype branch); now a
phm branch prints entries
Added spec cases (maps/collection-keys, sets/literals & value elements).
conformance 218/218, jpm test green.
Final spec areas. Bugs caught and fixed:
- (def name docstring value) used the docstring as the value; now the 3-arg
docstring form binds the value and records :doc meta
- resolve was a nil stub; now a special form resolving a symbol to its var
(nil if unresolved). Added find-ns (non-creating lookup) and ns-name.
- in-ns didn't evaluate its arg, so (in-ns 'foo) failed; now evaluates it per
Clojure (the integration test's unquoted form updated to the quoted idiom)
- #(... %& ...) built %& as a positional param instead of a & rest param;
now emits (fn* [... & gen] ...) so %& captures the rest
Full public-API spec layer now in place. conformance 218/218, jpm test green.
Add spec suites for stateful refs (atoms/volatiles/delays/promises), multimethods
(dispatch + hierarchies), and protocols/types/records. Bugs caught and fixed:
- reify with multiple methods registered only the first (stepped i by 2 over
one-element specs); now collects every method spec
- get-method/methods/remove-method evaluated their arg to the dispatch fn, but a
multimethod's methods live on its var — now resolve the var (get-method and
methods are special forms; remove-method fixed). get-method/methods added.
- no global hierarchy: 2-arg derive modified a throwaway, isa? was hardcoded
false, parents missing, ancestors/descendants returned arrays (so contains?
failed), and dispatch ignored derive. Now a global hierarchy backs the 1/2-arg
derive/isa?/parents/ancestors/descendants (returning sets) and multimethod
dispatch falls back to it.
Corrected a phase5 port assertion that encoded the old isa? bug.
conformance 218/218, jpm test green.
Add spec suites for conditionals/logic/let/loop/iteration/threading, functions
(definition, application, combinators), and destructuring across let/fn/loop/
doseq/for. Bug caught and fixed: loop* assumed simple-symbol bindings, so
(loop [[a b] [1 2]] ...) errored; it now destructures each binding via
destructure-bind, supporting patterns in loop bindings (and recur). jpm test green.
Add spec suites for numbers/arithmetic, strings (str + clojure.string), and
type/value predicates. Bugs they caught and fixed:
- <, >, <=, >= were binary-only; now variadic ((< 1 2 3) chains)
- seq? was true for vectors; vectors are not ISeq in Clojure, so (seq? [1])
is now false (true only for lists/lazy-seqs)
- clojure.string/index-of and last-index-of were 1-based (stray inc); now
0-based per Clojure
Corrected a systematic-coverage assertion that encoded the old seq? bug.
jpm test green, conformance 218/218.
Add behavioral spec suites for the collection API (~180 cases). Writing them
surfaced and fixed real bugs:
- (count nil) errored; now 0 (Clojure semantics)
- (repeat x) and (repeatedly f) — the infinite 1-arg arities — were unsupported;
now return lazy infinite seqs
- set equality used deep= (representation-sensitive), so a set of map literals
could differ from an equal set built differently; now value-based (each
element value-equal to some element of the other)
Known limitation filed: phm-typed maps (from hash-map) used as set elements /
map keys hash by identity (map *literals* are structs and work).
jpm test green.
The last two keystone forms in namespaces.cljc now load, so the entire SCI
source in the harness's load order evaluates with ZERO failures (was 420/2).
Two real bugs fixed to get there:
- with-meta crashed on non-collections: (with-meta a-fn {...}) did (keys fn) ->
'expected iterable, got function'. SCI marks core fns private this way. Now
returns functions/scalars unchanged (Jolt can't attach meta to a raw fn).
- select-keys ignored a vector key-list: (each k ks) iterated the pvec's table
instead of its elements, so (select-keys m [:a]) returned {}. Now realizes ks.
Plus: add clojure.set/join (the only missing clojure.set fn), assert 0 failures
in test-load-sci so it's a real regression guard, drop debug instrumentation.
Jolt now fully loads SCI's reader-independent implementation: every pure-Clojure
module, the entire clojure.core aggregation map, and the clojure.* namespace
wiring. conformance 218/218, features 78/78, jank 120.
