clojure.zip was missing xml-zip — a zipper over xml {:tag :content} elements,
which clojure.data.zip and any xml-zipper code needs. Added (runtime, loaded on
require). clojure.data.zip's whole xml suite (9/9) then passes, once XML parsing
is provided: clojure.xml/parse now ships in jolt-lang/xml over its
javax.xml.stream pull parser (committed there).
Listed in docs/libraries.md + the site.
clojure.core.contracts (over core.unify) now runs its whole suite on jolt —
14/14 across contracts/constraints/with-constraints/provide tests. Two general
gaps fixed:
- Symbol and Keyword now report IFn (and Fn/Runnable/Callable) in the modeled
class hierarchy, so a (class x)-dispatched multimethod with an IFn method
matches a symbol or keyword, like the JVM (both implement IFn — they're
callable). core.contracts' funcify* dispatches on (class constraint) and a
bare predicate symbol must hit the IFn arm. Runtime, no re-mint.
- A live Var value spliced into a form by a macro (defcurry-from resolves a var
and emits (~v l r)) now compiles: analyze treats a var-cell form as a
:the-var reference by ns+name, the same node as (var ns/name), mirroring the
existing spliced-namespace (~*ns*) case. analyzer.clj + host-contract.ss,
re-mint (prelude stays byte-identical; only the analyzer image changes).
Listed in docs/libraries.md + the site.
make test green (+2 corpus rows, 0 new divergences), shakesmoke byte-identical.
Adds clojure.core.async's higher-level dataflow API as a Clojure overlay
(stdlib/clojure/core/async.clj) over jolt's native channel primitives, plus
clojure.core.async.lab. The native layer (host/chez/java/async.ss) gains
offer!/poll!, put specs and :priority/:default in alts!, a transducer
ex-handler arg to chan, unblocking-buffer?, promise-buffer, and on-caller?
handling for put!/take!. The overlay covers alts!/pipe/pipeline/split/
reduce/transduce/into/take/mult/mix/pub-sub/map/merge/onto-chan/to-chan and
the deprecated map</map>/filter>/... family (rewritten as go-loops since the
JVM versions reify the impl handler protocol jolt doesn't expose).
Loading: the native primitives pre-seed clojure.core.async, so the loader now
drops it from the loaded set and a require pulls the overlay from the source
roots like clojure.test (AOT-bundled into built binaries).
Running clojure/core.async's own suite shook out two general bugs:
- :refer with a list form, (:require [ns :refer (a b c)]), dropped the names
(only the vector form was handled) — chez-register-spec! now accepts both.
- (range 0) / (range 5 5) returned nil instead of the empty seq () — empty
ranges now match Clojure, so (= () (range 0)) holds.
Suite: async_test 15/20, pipeline_test 7/7, timers_test 2/2, lab_test 2/2.
The five non-passing async_test cases all assert JVM go-machine limitations
jolt's thread-based model is a superset of (the 1024 pending-op cap, parking
ops that must throw outside a go block, expanding-transducer buffer
backpressure) or dispatch-thread identity, not data semantics.
make test green (0 new divergences, +4 range corpus rows), shakesmoke
byte-identical.
Both printers (jolt-pr-str, jolt-pr-readable) now thread a print depth and
read the two limit vars. *print-length* truncates each collection to N
elements + "...", walking seqs lazily so an infinite seq prints under the
limit without realizing it. *print-level* renders a collection at depth >=
the level as "#". The reader consults *default-data-reader-fn* for an
unregistered #tag before falling back (tagged form on the data seam, throw
on the edn seam). All three interned with nil defaults.
Spec coverage dashboard had 6 missing-portable and 24 dynamic-var entries. The
portable ones are now implemented (missing-portable -> 0, dynamic-var -> 14):
- Stateful matcher: re-matcher now returns a real mutable Matcher; re-find over
it steps through matches and re-groups returns the last match's groups (was an
inert tagged map). Closes re-groups.
- letfn is interned as a clojure.core var so (resolve 'letfn) matches the JVM. It
stays a special form (the value is never invoked, not marked a macro).
- *1 *2 *3 *e interned (nil outside a REPL).
- Portable dynamic vars whose default already matches jolt's behaviour:
*read-eval* *print-dup* *print-namespace-maps* *flush-on-newline*
*compile-files* *math-context* *command-line-args* *file*.
The remaining 14 dynamic-var entries are host-internal (compile-path,
compiler-options, fn-loader, reader-resolver, repl, source-path, ...) or deferred
pending printer/reader support (*print-length* *print-level*
*default-data-reader-fn*). Corpus rows added for each closed gap; coverage.md
regenerated.
irregex rejected two patterns the JVM accepts, which blocked library loads:
- [\w-_] errored with bad char-set because a - after a shorthand class was
read as a range start. Java reads it as a literal hyphen. Preprocess the
pattern to escape such a dash.
- (X+)* errored with duplicate repetition because sre-repeater? recurses
through submatch, treating a quantified group like a dangling a**. Override
it to a bare leading * / + check, matching the JVM (which only rejects the
dangling case).
Both in regex.ss (runtime). Unblocks cuerdas (was load-fail, now 292 passing)
and aws-api config-test. Also documents the host/chez/java source-layering rule
in host-interop.md.
jolt-l8so
A type-aware audit (~190 collection expressions vs reference Clojure) found four
divergences the corpus missed — value-equality (= [0 1] '(0 1)) hides type and
laziness differences. Fixed, with type-predicate + over-infinite corpus rows that
pin them.
- partition-all [n coll] built vector chunks; JVM chunks are seqs. (The [n step
coll] arity was already correct, as is the partition-all transducer, whose
chunks are vectors in JVM too.) Now builds seq chunks.
- replace always returned a vector (mapv) and was eager; JVM is type-preserving —
a vector maps to a vector, any other seqable to a lazy seq.
- sequence eagerly realized its source (into-xform), so (first (sequence (map inc)
(range))) hung. Rewrote as a transformer iterator: pull one input at a time,
buffer the step outputs, emit lazily, run the completion to flush a stateful
xform. eduction builds on it (lazy, no longer an eager vector).
- mapcat and (apply concat coll-of-colls) hung over an infinite source because
jolt-apply seq->lists the trailing arg and mapcat seq->lists the map result.
Added lazy-concat-seq (lazily flatten a seq of colls); mapcat uses it directly,
and apply special-cases concat (its result is lazy) to route through it.
Docs: a cross-cutting return-type + laziness contract in docs/spec/09-core-library;
SPEC.md notes that = masks type/laziness so they need predicate / over-infinite
rows. EBNF is reader syntax only — unaffected.
Seed change (partition-all/replace/eduction are clojure.core overlay) -> re-mint;
selfhost holds. make test + shakesmoke + buildsmoke green, 0 new divergences.
Co-authored-by: Yogthos <yogthos@gmail.com>
Shaking out clojure.core.memoize (207 assertions, 0 fail) cleared several
general gaps:
- deref/@ on a deftype or reify implementing clojure.lang.IDeref dispatches to
its deref method (RetryingDelay / make-derefable).
- deftype mutable fields (^:unsynchronized-mutable / ^:volatile-mutable) are
read live: a set! within a method is observed by a later read in the same
invocation, not the entry-time capture. Needed for double-checked locking.
Immutable fields stay let-bound. Field reads rewrite to (.-field inst) with
lexical-shadow tracking.
- def metadata values are evaluated, like Clojure: ^{:k (f)} stores (f)'s
result and ^{:af some-fn} the fn. :tag stays a literal hint.
- try dispatches catch clauses by class in order via the exception supertype
hierarchy; a non-matching value re-throws, an untyped host condition is caught
by a RuntimeException/Exception/Throwable clause. Previously the last clause
won and the class was ignored.
- locking takes a real per-object monitor (recursive Chez mutex) now that
futures/agents/threads share one heap; it was a no-op.
- supers/ancestors reflect a small modeled JVM interface hierarchy, so
(ancestors (class f)) yields Runnable/Callable (core.memoize's arg check).
- AssertionError / Error constructors.
JOLT_FEATURES is gone from the docs: it isn't read anywhere on Chez, and the
reader already includes :clj in its default feature set. RFC 0002's
{:jolt :default} design was reverted in the reader; docs now match the code.
Raises the SCI floor 205 -> 210.
Finishes core.match — its full test suite (115/115) now passes, including the
two patterns the earlier work left out:
- Regex-literal patterns. A #"…" now reads as a regex VALUE (Clojure parity: the
reader constructs the Pattern, so a macro receives a regex, not jolt's tagged
form), and the analyzer compiles a regex value to the same :regex IR leaf via
its source. emit-quoted handles a quoted regex; a regex value carries the
java.util.regex.Pattern host tag so extend-protocol/instance? dispatch on it.
- Primitive-array patterns. A ^Type hint's :tag is now the SYMBOL (e.g. `ints`),
matching the JVM, so core.match's array-tag lookup engages the array
specialization (alength/aget). jolt's :tag consumers already tolerate a symbol
(hc-cell-num-ret normalizes; tag->nkind/def-meta handle both).
Also: a library-conformance directive in CLAUDE.md, and the supported-libraries
list (docs + site) simplified to one-line entries — a listed library is assumed
to work fully, so no tallies or feature enumerations. core.match + transit-jolt
added to the list.
Seed change (reader/backend/30-macros) -> re-minted; the rest runtime. JVM-
certified corpus rows; the stale `symbol hint -> :tag` divergence is dropped from
the allowlist (jolt now matches the JVM). make test + shakesmoke green.
Expand java.io so libraries that touch the filesystem work unchanged.
File: the full method surface — length, lastModified, can{Read,Write,Execute},
isHidden, list, mkdir(s), delete, createNewFile, renameTo, getParentFile,
get{Absolute,Canonical}File, compareTo/equals/hashCode — plus the statics
separator / pathSeparator / createTempFile / listRoots. A File now keeps the
path as given (new File("rel").getPath() is "rel", .isAbsolute false); a
relative path resolves against JOLT_PWD only when the filesystem is touched,
matching the JVM. slurp/spit and the dir helpers go through the same
resolution, fixing a spit-vs-slurp inconsistency.
Streams (host/chez/io-streams.ss) — each a jhost wrapping a Chez port, so
buffering, EOF and binary<->char transcoding come from Chez:
- FileInputStream / FileOutputStream / ByteArrayInputStream /
ByteArrayOutputStream / BufferedInputStream / BufferedOutputStream
- FileReader / FileWriter / InputStreamReader / OutputStreamWriter /
BufferedReader / BufferedWriter
Buffered* return the wrapped stream (Chez ports are already buffered).
clojure.java.io: input-stream/output-stream now yield real byte streams (were
aliased to the char reader/writer); added copy (byte-exact for byte sources),
make-parents, delete-file. with-open also closes file-writer/port-writer/
print-writer (a pre-existing gap).
All runtime shims, no re-mint. 15 JVM-certified corpus rows; make test +
shakesmoke green.
Shaking out tick's api and alpha.interval suites (api 353->359, interval
0->103 passing) cleared a set of general gaps:
- Named-zone DST. Zones resolved to a fixed representative offset, so
America/New_York in August read -05:00 not -04:00. Add US/EU DST rules
(compact transition-date math) and make instant<->zoned, the zone rules'
getOffset, and the zoned equality arm DST-aware.
- Nanosecond zoned/offset times. Instant is nanos but atZone/atOffset/
toInstant/withZoneSameInstant and Instant/parse went through epoch-ms,
truncating sub-ms. Route them through nanos.
- Locale month/day names. A formatter dropped its Locale; carry it and add
French names so MMM under Locale/FRENCH renders "mai".
- Callable records. A defrecord implementing clojure.lang.IFn (tick's
GeneralRelation) is now invokable: jolt-invoke dispatches to its inline
invoke method. Also give collections the Iterable host tag so a protocol
extended to Iterable matches vectors/seqs.
- Imported class short names. (:import [java.time ZonedDateTime]) then
(. ZonedDateTime parse s) resolved to nil; an otherwise-unresolved bare
Capitalized name that's a registered host class now resolves as a class.
- Data readers. A project's data_readers.{clj,cljc} is loaded into
*data-readers* (reader namespaces required eagerly); registered #tag
literals in source rewrite to (reader-fn 'form). clojure.core/read-string
now applies #inst/#uuid/#"regex" and *data-readers* like Clojure.
- Duration/between accepts zoned/offset date-times.
All runtime shims, no re-mint. docs/libraries.md: tick full pass + aero.
The EBNF and reader S7 already specified ::kw auto-resolution — the
implementation was out of spec, now aligned. S7 noted unknown ::alias/k MUST be a
read error; jolt is lenient (reads :alias/k), so record that as a deviation.
Corpus gains JVM-certified rows for ::kw resolution, clojure.edn :default
receiving a symbol tag, and with-meta on a lazy seq.
The source loader sets the current ns and registers :as aliases per file. The
build flattened every app namespace into one image with no such markers, so all
app forms ran under the last-set ns ("user"). Two breakages followed, both only
in a built binary:
- defmulti/defmethod resolve their target var through chez-current-ns, so they
registered the multifn under "user" while compiled var-derefs used the baked
ns — the multifn the app saw was uninitialized ("not a fn nil" on dispatch).
- a quoted alias-qualified symbol (a (defmethod ig/foo …) on an aliased multifn)
resolves its ns through chez-resolve-alias, but the stripped (ns …) form left
the alias table empty, so it landed in ns "ig".
bld-ns-prelude now emits (set-chez-ns! ns) plus chez-register-alias! for each
ns's :as aliases before that ns's forms, in both the normal and tree-shake emit
paths. The build-app fixture gains a :default multimethod and an aliased cross-ns
defmethod so buildsmoke covers both across all build modes.
Two small clarity moves from the review:
- dynamic-vars.ss -> dynamic-var-defaults.ss. It holds the default VALUES of a few
core dynamic vars (*clojure-version*, *assert*, …); the near-identical dyn-binding.ss
holds the binding-stack machinery. The names were easy to confuse.
- Pull the ~60-line %-format engine out of the natives-misc.ss grab-bag into
natives-format.ss (its ->long/pad-left/fmt-float helpers were local to it).
rt.ss loads + MODULES.md updated. Runtime .ss, no re-mint; make test green, format +
the dynamic vars verified.
The trio split on a fine axis (registry core / statics / object classes) but the
names didn't say so — 'static-statics'/'static-objects' and headers that read
'Continues X'. Rename:
host-static-statics.ss -> host-static-methods.ss (Class/member statics + fields)
host-static-objects.ss -> host-static-classes.ss (instantiable object classes)
host-static.ss stays the registry core. Headers rewritten to state each file's role
and what it covers instead of chaining. rt.ss loads + the one comment reference +
MODULES.md updated. No code moved; runtime .ss, make test green.
Navigability groundwork from the architecture review — zero behaviour change.
- docs/MODULES.md: the repo map. Area -> directory -> key files -> re-mint?, plus
per-feature touch points (tree-shaking, direct-linking, numeric fl/fx, inlining,
multimethods, deps) and where a given clojure.core fn lives. Answers "where does
X live / what's related to Y" in one read.
- docs/rfc/README.md: index the 7 RFCs; flags RFC 0007's stale "no code yet" status
(direct-linking + tree-shaking shipped) and the undocumented inlining/numeric work.
- CLAUDE.md: document the var-deref calling convention (public defns reached from the
.ss runtime by string lookup aren't dead), the def-var! native pattern, and the
overlay shadowing rule; point at MODULES.md.
- REFACTOR_PLAN.md: the prioritized, risk-tiered plan (working doc for this branch).
README + tools-deps.md cover --tree-shake and --direct-link: what tree-shaking does
(whole-program reachability over app + libraries + clojure.core, drop unreachable,
drop the compiler for no-eval apps), and why it bails to keep-all when reachable code
resolves vars by name at runtime (eval/resolve/ns-resolve/...), with the diagnostic
output and how to make an app shakeable. Notes the Stalin soundness model.
Release builds can legitimately want runtime dynamism (redefinition, eval,
load-string), so closed-world direct-linking shouldn't be forced on them. Gate it
behind an explicit --direct-link flag (or deps.edn :jolt/build {:direct-link
true}); off by default in every mode, including release and --opt.
build-binary takes an explicit direct-link? arg instead of deriving it from the
mode. build-smoke now covers the --direct-link path and asserts the cross-ns call
actually lowers to a jv$ binding; default release stays dynamically linked.
Build output landed in the CLI's cwd (the jolt repo, since bin/joltc cd's
there), not the project — so a bare -o path or the default binary appeared
in the wrong place. Resolve output against JOLT_PWD, and default it cargo-
style under the project's target/: target/release for release/--opt,
target/debug for --dev, named after the project dir. The <name>.build scratch
dir sits beside the binary, so it lands under the same target dir. -o is
honored — absolute as-is, relative against the project.
A built binary dropped its deps.edn :jolt/native declarations and its
resource roots, so an FFI+resources app (ring-app) failed at runtime:
sockets/sqlite gave 'no entry for socket' and io/resource returned nil.
The buildsmoke fixture is pure compute, so neither path was exercised.
The launcher now loads required + :process native libs before the app's
top-level forms (a library's defcfn resolves its foreign-procedure symbols
at top-level eval during startup, so the libs must be loaded first);
optional libs load in the scheme-start launcher, where a missing lib is
caught rather than aborting the heap build.
deps.edn :jolt/build {:embed [dirs]} bakes those dirs' files into the heap
(register-embedded-resource! at heap build), so io/resource serves them with
no files on disk. Non-embedded resources resolve at runtime against JOLT_PWD,
and io/file reads (e.g. config.edn) stay external.
build-binary now takes the encoded natives, embed dirs, and project paths
from cmd-build; deps/resolve-project surfaces them. Buildsmoke fixture grows
an embedded resource + a :process native to cover both paths.
Restores the standalone-binary capability the Janet host had. `bin/joltc build
-m NS -o OUT` AOT-compiles an app into a single self-contained executable — the
whole runtime, clojure.core, stdlib and compiler embedded, no Chez install or
jolt source needed at runtime.
Pipeline (host/chez/build.ss, host primitive jolt.host/build-binary driven by
jolt.main's build command): resolve deps, load the entry namespace recording the
app namespaces in dependency order, re-emit each to Scheme, textually inline the
cli.ss runtime load sequence into one flat source + the app + a launcher, then
compile-file -> make-boot-file -> embed the boot as C bytes -> cc-link against
libkernel.a.
Two non-obvious bits: the compile pass runs in a fresh Chez, not the loaded
runtime (regex.ss shadows top-level `error`, which otherwise bakes a broken
reference into the boot); and the launcher installs scheme-start rather than
running -main at top level, since boot top-level forms execute during heap build
before argv is set, so args only reach -main through scheme-start.
Loader: a require of an in-memory namespace with no source file now no-ops, so
AOT'd app namespaces satisfy require in a built binary.
Mode flags (--opt/--dev, default release) are plumbed; the optimization passes
they gate come in a later stage. RFC 0007 has the design. Gated by `make
buildsmoke`.
Rename src/jolt -> stdlib (the runtime-loaded layer; jolt-core stays the
seed-baked layer) and update the loader / emit-image / doc paths. Drop dead
code: the spike/ experiments, the duplicate clojuredocs-export.edn (json moves
to tools/), the Janet-era jolt.http binding, and the orphaned
persistent_vector.clj whose ns/path didn't even match.
Strip porting residue from comments and docstrings across host/chez, jolt-core,
stdlib, tests, and docs: internal issue ids, "Phase N" markers, and the "vs
Janet" historical exposition, leaving present-tense descriptions and the real
JVM-Clojure semantic contrasts. Same pass over the corpus suite labels. The seed
is unchanged (docstrings/comments aren't emitted), so the self-host fixpoint and
corpus are untouched.
Port tools/spec_coverage.py off the dead janet probe to bin/joltc and regenerate
coverage.md; drop the dead :host/janet rule from certify.clj and regenerate the
conformance profile. Add docs/host-interop.md (the JVM shims and how to register
your own host class from a library) and a writing-style note in CLAUDE.md.
Stabilize the four racy concurrency corpus cases (future-cancel and agent
send/send-off): give the future a sleeping body and the agent a slow action, so
cancel reliably catches an in-flight future and deref reliably reads the
pre-update snapshot. They certify deterministically now, so drop their :flaky
allowlist entries and the orphaned legend.
The line REPL was broken (read-line called nil — the __stdin-read-line host seam
the clojure.core *in* reader drives was never implemented on Chez) and didn't
load the project, so (require '[some.lib]) failed. Now:
- __stdin-read-line reads a line from stdin (get-line); read-line / read / the
REPL work.
- repl resolves the project first (deps on the roots, native libs loaded), so
libraries are available — same context a run gets.
- jolt.nrepl: a jolt-native nREPL server (bencode over a loopback jolt.ffi
socket) speaking the real protocol — clone / describe / eval / load-file /
close, with stdout capture, :ns-scoped eval (in-ns; binding *ns* doesn't drive
load-string resolution here), and real error text. 'joltc nrepl [port]' applies
the project then serves; writes .nrepl-port. Editors (CIDER/Calva/Cursive)
connect and develop live; project libraries load in the session.
- ex-message returns nil for raw Chez conditions, so jolt.host/condition-message
exposes the condition text; the REPL and nREPL surface it instead of an opaque
#<compound condition>.
Why native, not real nREPL: nrepl.server is welded to java.util.concurrent
executors, two compiled Java helper classes, a DynamicClassLoader, Compiler
internals and a JVMTI agent — not faithfully shimmable. The wire protocol, which
is what clients depend on, is small and implemented directly.
Runtime .ss + jolt-core, no re-mint. Full gate green.
Bring the docs in line with the actual implementation now that Chez is the sole
substrate.
Deleted the migration/spike/handoff artifacts that only documented the Janet
era or the port effort: the port plan, phase-0 and foundational-runtime spike
writeups (+ the stray root-level copy), the self-hosting design notes, the
architecture-refactor plan, and spike/chez/RESULTS.md.
Rewrote the current reference docs against the Chez facts: building-and-deps and
tools-deps (no jpm/build step — bin/joltc off the checked-in seed, deps via
jolt.deps into ~/.jolt/gitlibs), libraries (SQLite is built-in jdbc.core over
libsqlite3, not a Janet driver), the conformance/spec test-flow docs (the Chez
corpus runner + certify, no .janet harnesses), and the transient / type-hint /
seed-overlay design notes (Chez representations: mutable transients, flat
copy-on-write vectors, HAMT maps, the seed/overlay twin). Fixed the README
collections line (vectors aren't 32-way tries) and added the ffi/transient gate
targets. rfc 0001's numerics open-question is resolved (the Scheme tower).
Renamed the built-in HTTP adapter to jolt.http.server only (dropped the
ring-janet.adapter alias — a Janet-era name).
Bring the formal definition in line with this session's language work:
- grammar.ebnf: numbers are a real tower (exact integer / Ratio / double); the M
suffix reads a real BigDecimal, N an exact integer (drop the stale Janet note).
- 02-reader S5: M is a real java.math.BigDecimal with scale-insensitive equality.
- 03-special-forms: document the read -> macroexpand -> analyze order (macros
expand before special-form dispatch); special-form heads are not shadowable but
macros are and value-position locals may be named like a special; set! on a var
sets the innermost binding (else root); letfn is a primitive with letrec*
semantics.
Revise the epic's direction from a minimal Janet shim to ripping Janet
out entirely — Chez becomes the sole substrate. The missing spine: the
compiler pipeline itself only runs on Janet today (the analyzer executes
on the Janet host; the IR->Scheme emitter is host/chez/emit.janet). Phase
3 is re-scoped to self-host the compiler on Chez (emitter -> Clojure
jolt.backend-scheme, reader -> jolt-core, compile-from-source bootstrap
fixpoint). Phase 5 becomes a hard delete of both src/jolt/*.janet and
host/chez/*.janet. Sequencing: core parity first, then self-host, then
delete.
0c: persistent HAMT on Chez is ~41x faster than Janet's HAMT on the collections
map-churn (258.6 -> 6.3 ms), ~15x off mutable-native (inherent persistence cost).
Decision: self-host the persistent collections in Clojure; substrate is not the
bottleneck. See docs/chez-phase0-results.md.
0a hardening: NUL-separated keyword intern key (no ns/name collision), non-finite
-safe jolt-hash. 37/37.
host.* replaces janet.* as the portable interop namespace (each host implements
it over its own FFI). Add a :native dep form so projects declare needed shared
libs (libcurl/openssl/zlib) — not git-fetchable, but surfaced to the user and
probed at load so a missing .so yields a precise error, not a raw dlopen fail.
Account for jolt's layered interop surface on Chez — the janet.* bridge, the
FFI-backed java.* shim libs (http-client TLS/gzip, router, db), jpm-module Janet
deps (spork/http) — with ../examples as the end-to-end acceptance gate. New
epic child jolt-cf1q.7, gated behind Phase 2.
Phased plan for re-hosting jolt's substrate on Chez Scheme, organized around two
north stars: minimal host shim (push everything possible into self-hosted
jolt-core, drop the tree-walking interpreter) and the spec/conformance corpus as
the host-neutral correctness contract. Closes obsolete Janet-backend/cgen beads
superseded by the native substrate.
Fuse an app's native-compiled numeric-leaf fns plus its source into one
static executable: no sidecar .so, no toolchain on the target. The AOT path
(#148) already produced a prebuilt module + manifest; this links them into
the jpm-built exe so the app ships as a single file.
`jolt cgen-build -m NS -o OUT` stages a build dir (src/jolt-core symlinks
into the jolt tree, a generated cg.c of the hot fns, an uberscript bundle of
the app, and an entry that bakes the runtime, installs the native fns as var
roots, and runs -main), then runs `jpm build` there — declare-native builds
cg.a and declare-executable static-links it (jpm's create-executable marshals
the module cfns and calls its static entry at startup).
Build needs cc + jpm; the result needs neither. Mechanics that bit, codified
in cgen_build.janet: stdlib_embed slurps .clj cwd-relative so the build runs
in a repo-mirroring dir; jpm hardcodes ./project.janet and sets syspath=modpath;
the executable's dofile imports cg and static-links cg.a, neither ordered nor
release-built by default, so deps are wired explicitly; cleanup must lstat (the
tree symlinks must not be followed); the inner build runs --workers=1 so it
doesn't starve siblings in the parallel gate.
test/integration/cgen-build-test.janet builds the mandelbrot fixture, runs it
from a clean dir with no src/ and no cg.so, and checks the total at native
speed. Closes jolt-a7ds.
Co-authored-by: Yogthos <yogthos@gmail.com>
Lever 1 (native codegen) is built and merged (PRs #143-148): the floor is
localized, cgen translates numeric-leaf fns to C (JOLT_CGEN, 18x on mandelbrot,
cached), and the build-time AOT path deploys native code with no cc. Replaces the
stale START HERE (which still pointed at the now-done spike) with current status
and the open work: jolt-a7ds binary fusion, jolt-v28u while-lowering, jolt-l1l4
grammar widening, jolt-qx70 hot-fn detection.
Co-authored-by: Yogthos <yogthos@gmail.com>
Splits native codegen into a build phase (needs cc) and a deploy phase (none):
- gen-c-module/compile-module compile MANY numeric-leaf fns into ONE native
module (the AOT shape), generalizing the one-fn-per-.so JIT path.
- Backend :cgen-collect? records each numeric-leaf defn's IR while the app loads
as bytecode; cgen/aot-build compiles them into one module and write-manifest
persists {sopath, [{ns name sym}]}.
- Backend :cgen-prebuilt + cgen/load-aot: the deploy run loads the prebuilt .so
(via the native builtin, no cc) and installs each cfunction as the var root
with the same timing as the JIT path, so callers direct-link to native code.
- toolchain-available? no longer crashes when cc is off PATH (os/execute raises
on a missing exe) — a toolchain-less target now gets false.
Proven end-to-end in two processes (spike/native/aot-demo.janet): build with cc,
then deploy with cc removed from PATH -> count-point still native, mandelbrot
3288753 at 12.4ms (full 18x). Test: test/integration/cgen-aot-test.janet. Default
path unchanged; the modes are opt-in. Gate green (118 files).
Remaining for a literal single binary: fuse the .so + manifest into the jpm exe.
Co-authored-by: Yogthos <yogthos@gmail.com>
Wires src/jolt/cgen.janet into the backend's :def emit. With JOLT_CGEN=1 (off by
default, needs direct-linking), a plain defn of a numeric-leaf fn is compiled to
C at def time and the cfunction installed as the var root, so direct-linked
callers embed native code. The fn is not inline-stashed when cgen fires —
callers must call the C fn, not inline the bytecode body. ^:redef/^:dynamic stay
bytecode.
The leaf-first rule falls out: run calls count-point (a user var), so run isn't a
numeric leaf and stays bytecode, calling the native count-point over the cheap
forward crossing. mandelbrot 200: 224ms -> 12.4ms (~18x), result unchanged.
Adds JOLT_CGEN to ctx-shaping-env-vars (rides the disk-cache key) and :cgen? to
resolve-run-mode. Default path (cgen off) is a no-op: cgen-root returns nil and
the normal bytecode emit runs. Gate green (117 files). Test:
test/integration/cgen-pipeline-test.janet.
Co-authored-by: Yogthos <yogthos@gmail.com>
Probes the ceiling and incremental strategy for compiling hot fns to native C,
the only lever that moves the ~10.8x Janet-VM floor the localization spike found.
Native-C mandelbrot (Janet native module) runs ~10-12ms — faster than JVM
Clojure (14.2ms) and ~18-22x faster than jolt's 219ms. The boundary cost is
asymmetric: a bytecode loop calling a C hot-fn 40k times is nearly free (~11ms),
but a C fn calling back into bytecode via janet_call costs ~3.5us/call (~152ms,
no win). So the strategy is leaf-first / whole-hot-cluster compilation, crossing
only at cold edges. A plain cc-built .so (no jpm) loads at runtime via require at
full speed, so the native tier fits jolt's dynamic compile model.
Adds the spike artifacts under spike/native/ and the writeup. Next step is
jolt-ihdp (IR->C for the numeric subset). No source changes.
Co-authored-by: Yogthos <yogthos@gmail.com>
The jolt-vs-hand-Janet-vs-JVM mandelbrot comparison splits the 15.4x floor
into two layers: a Janet-VM floor (~10.8x JVM, optimal while-loop Janet over
unboxed doubles — only native codegen moves it) plus a ~1.43x jolt loop-
lowering overhead on top. The overhead is entirely the loop/recur -> recursive-
closure-called-per-iteration lowering; hand-Janet written the same way matches
jolt, while a while+var/set version is 1.43x faster. So a cheap backend win
(jolt-v28u) sits above the structural native-codegen lever.
Adds the spike artifacts under bench/ and the results writeup; marks the spike
done in the handoff. No source changes.
Co-authored-by: Yogthos <yogthos@gmail.com>
The targeted-specialization work (jolt-ffn) concluded that the constant-factor
gap vs JVM is structural, not per-form: three targeted passes (field-read,
inline cache, ctor descriptor-bake) all came back flat. mandelbrot (pure
compute) is ~15x off JVM and that's the floor — Janet bytecode VM + mark-sweep
GC + indirect calls.
This doc hands off the successor epic (jolt-5vsp): the foundational levers
(native codegen, GC-pressure reduction, deeper devirt+inline) and, importantly,
the spike to run first — localize the 15x floor by comparing jolt-compiled vs
hand-written-Janet vs JVM mandelbrot before committing to any big lever. Also
records what not to repeat.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Bind *command-line-args* after the deps-image cache swap (jolt-4mui)
Under whole-program (deps-image cache active), `jolt -m NS ARG` dropped ARG:
run-main set *command-line-args* on the current ctx, but a cache HIT then
replaced ctx with the saved image (via `set ctx cached`), whose *command-line-
args* was whatever got baked when the image was saved. The stale binding won at
`(apply NS/-main *command-line-args*)`, so -main ran with the wrong (usually
default) args — silently, for any optimized -m program.
Move set-command-line-args to AFTER the cache swap so it binds on the final ctx.
Repro/regression in deps-cache-args-test.janet: first run builds the image
(arg "first"), second run (cache hit) must echo "second", not the baked "first".
* docs: RFC 0003 — phm is a HAMT, sorted colls a red-black tree
The transients RFC described phm as "bucket-based copy-on-write" and mused about
"if it ever becomes a HAMT" — it is one now (jolt-684u), and sorted maps/sets are
a red-black tree (jolt-0hbr). Update the deviation/future-work notes accordingly.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
Dependency resolution now lives in the `jolt` CLI itself instead of a separate
jolt-deps executable. `jolt` resolves a deps.edn into JOLT_PATH/JOLT_APP_PATHS
in-process and dispatches the deps subcommands:
jolt -M:alias [args] run the alias :main-opts
jolt -A:alias CMD run CMD with the alias paths
jolt run FILE resolve, then run FILE
jolt path | tasks | task NAME
A deps.edn in the working dir is auto-resolved for the runnable commands
(repl/-m/-e/nrepl-server/FILE), so e.g. `jolt -M:nrepl` (or plain
`jolt nrepl-server`) starts an nREPL with the project and its deps loaded.
The runtime core stays deps-agnostic — it only reads JOLT_PATH. The resolver
(deps.janet) is reached only from the CLI entry and loads jpm lazily, so a run
with no deps.edn never touches it and an app baked from its own jolt/api entry
never links it. resolve-deps-argv only resolves on an explicit deps command or
when a deps.edn is present; help/version never do.
jolt-deps stays as a thin deprecation shim that forwards to `jolt`, so existing
scripts keep working. Docs (README, CLAUDE.md, building-and-deps, tools-deps)
and the help text updated.
Co-authored-by: Yogthos <yogthos@gmail.com>
* Reader: #() params survive syntax-quote (auto-gensym names)
#(...) named its synthesized params with bare gensyms, so a #() written inside a
syntax-quote had its params qualified to the current ns by sq-symbol — and a
qualified symbol isn't a valid fn param. hiccup's compiler emits
`(let [sb# ..] (iterate! #(.append sb# %) ..)), which broke with "Unable to
resolve symbol: ns/_NNNN".
Name the params with a trailing # (auto-gensym suffix, like Clojure's p1__N#) so
syntax-quote maps them consistently and leaves them unqualified. Harmless outside
a backtick (just a regular symbol name).
* interop: String/valueOf static + String is a CharSequence
Two interop gaps surfaced bringing up hiccup and malli:
- String/valueOf(Object): hiccup's compiler stringifies attribute values with
(String/valueOf (or arg "")). Added the static — "null" for nil, else core-str.
- (instance? CharSequence s) returned false for a string; String implements
CharSequence, and malli's :re validator gates on it before matching, so :re
schemas always failed. instance-check now answers true for strings.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
