* core: compile macro expanders via staged bootstrap (Stage 2 Task 1)
Macros are now compiled, not interpreted, by steady state — matching
Clojure (macros are ordinary compiled fns the Java seed compiles for
clojure.core) and ClojureScript (macros compiled, invoked at compile
time). Neither reference keeps an interpreted-closure fallback.
The early macros (00-syntax) are defined while the self-hosted analyzer
is still being bootstrapped (it builds lazily after the overlay loads),
so macro-compile-hook returns nil and they get an interpreted closure.
The bootstrap compiler.janet can't substitute (it punts on syntax-quote,
which nearly every expander uses).
Fix = staged bootstrap, the same pattern as the compiler fixpoint:
defmacro stashes the expander source on the var (:macro-src) plus a
:macro-uses-env flag; once the overlay + analyzer are fully built,
backend/recompile-macros! (via ensure-macros-compiled! at the end of
load-core-overlay!) compiles each stashed expander through the now-live
analyzer and rebinds the var, marking :macro-compiled. Idempotent;
&env/&form macros keep the interpreted closure (the compiled fn* has no
such params). The interpreter is now a build-time crutch, gone by
steady state.
Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring
(which the analyzer punts on) to a plain rest param + (first rest), so
all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79.
47/47 overlay macros compiled, 0 interpreted; user macros compile
immediately post-init. Gate green: conformance 267x3, fallback-zero
31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap,
lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap,
features 78/78, all unit+spec, core-bench neutral.
* core: wrap macro expanders in fn (not fn*) so destructured arglists compile
Follow-up to the staged macro-compile change. The macro-recompile pass
wrapped each expander in the raw fn* primitive, which punts on a
destructuring rest param — so if-not/if-let/if-some/assert (using the
canonical '& [else]') couldn't compile and had been rewritten to plain
rest params as a workaround.
Root cause: fn* is the primitive; the fn MACRO is what desugars
destructuring (rest, map, nested) into the body before lowering. Wrapping
expanders in fn instead of fn* compiles any destructured macro arglist
uniformly, so the workaround is unnecessary — reverted those 4 macros to
the canonical '& [else]' forms.
Net: rest-destructuring is fully compiled for all normal code (fn/defn/
let/macro params). Only the hand-written raw fn* primitive still punts
(jolt-f79, downgraded to P4 — falls back to interpreter, still correct).
47/47 overlay macros compiled. Gate green: conformance 267x3,
fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host,
staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67,
sci-bootstrap, features, all unit+spec.
* core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79)
jolt-f79 asked to compile destructuring in fn* rest params. Checking
against Clojure inverts the premise: Clojure's fn* REJECTS destructuring
at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding
form, expected symbol'). So the self-hosted analyzer was already correct
— fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn
MACROS desugar destructuring. The real defect was the interpreter
leniently destructuring raw fn*, and defn emitting raw fn* to rely on it.
Changes:
- evaluator: fn*/let*/loop* now reject non-symbol binding forms with
Clojure's exact messages (require-symbol-params/plain-sym?), so the
interpreter agrees with the analyzer + Clojure.
- 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring
params desugar; unnamed, so self-recursion still resolves via the var.
- 00-syntax: completing that exposed a real gap — the overlay destructure
fn didn't handle kwargs (a map pattern bound against a fn's sequential
rest); it had only worked via the interpreter's destructure-bind. Added
the seq->map coercion to the map? branch (sequential: 1 map elt => that
map, else apply hash-map), matching destructure-bind so interpret ==
compile.
Net: fn*/let*/loop* are plain-symbol primitives across interpreter,
analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro
params, incl kwargs & {:keys}) compiles through the macros with no
interpreter fallback. Regression spec added.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44,
clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all
unit+spec (destructuring 50/50).
* docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79)
grammar.ebnf: the destructuring note already attributed patterns to the
binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES
they desugar to take plain symbols only (a non-symbol binding errors, as
in Clojure).
self-hosting-compiler.md: the 'compile destructuring via a shared
destructure expander instead of falling back' item is done (and its
jolt-7dl ref was stale) — destructuring now compiles through the
fn/let/loop/defn macros' desugaring; the primitives reject patterns.
* core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy
First slice of moving stateful forms onto the compile path (jolt-eaa).
- loader stateful-head?: drop syntax-quote (the analyzer's `handled` set
already compiles it; routing it to the interpreter was redundant).
- host_iface special-names: drop definterface/extend/proxy (stub macros
expanding to def/nil — their expansions compile once unpunted).
- letfn stays interpreted: its let* expansion needs letrec semantics
(mutual recursion between the fns), which sequential compiled let* lacks
— a later tier.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + protocol/multimethod/macro specs.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
adds self-hosted compiler is functionally:
- The default compile path is the portable pipeline using jolt.analyzer (Clojure) → host-neutral IR → backend.janet.
- The analyzer is itself Clojure, compiled by jolt for true self-hosting.
- bootstrap-fixpoint passes (stage1 == stage2 == stage3): rebuilding the compiler on its own output.
- clojure.core is now self-hosted in the overlay.
- Stateful forms (defmacro/ns/deftype/defmulti/require/in-ns) are interpreted by design.
Destructuring (drove by trying to load Selmer):
- defmacro params now destructure like fn (parse-params + destructure-bind), so
[& [a & more :as all]] and {:keys ...} work in macro arglists.
- map-destructuring a sequential value treats it as keyword args (alternating
k/v, or a trailing map) — the [& {:keys [...]}] form, in fns and macros.
- :keys/:syms accept namespaced symbols (x/y): looks up the namespaced key,
binds the bare local.
deps.edn: read with Jolt's reader instead of Janet's parse, so EDN ';' line
comments are handled (Janet treats ';' as splice).
clojure.java.io: the shim was at clojure/java_io.clj (ns clojure.java-io, never
the clojure.java.io that code requires) and used bare-qualified Janet calls that
the janet.* bridge no longer resolves. Moved to clojure/java/io.clj and rewritten
on janet.file/os: file/reader/writer/resource/copy/delete-file/make-parents.
Specs in destructuring-spec; destructuring note in the ebnf.
The reader expanded ^X form to (with-meta form X), which evaluated the tag (so
^String errored 'Unable to resolve symbol: String') and, as a param, was no
longer a bare symbol (so the arg bound to nil). Now a keyword/symbol/string hint
on a symbol attaches to the symbol's :meta and the symbol stays bare, so type
hints are transparent in params, lets, and bodies. Map metadata still uses a
runtime with-meta form.
meta now returns a symbol's :meta, and def applies the name's metadata
(^:dynamic, ^:private, ^Type tag, ^{:doc}) to the var, so (meta (var x)) is
consistent. Specs in metadata-spec; grammar note in the ebnf.
README notes regex \p{...} as unsupported (separate from this).
read-char read the char name with symbol-char?, so a literal whose char isn't a
symbol char (\{, \(, \), \,, \%, …) came back as an empty name and errored
'Unsupported character'. Now a single non-symbol char after \ is taken as a
one-character literal of that char. Surfaced by funcool/cuerdas (uses \{), which
now loads. Spec cases added.
The Clojure stdlib namespaces (clojure.string/set/walk/edn/zip, jolt.http/
interop/shell/nrepl) were loaded cwd-relative, so the built binary couldn't
load them outside the repo. Now stdlib_embed slurps them all at build time into
a {ns -> source} map frozen in the image, and the loader falls back to it when a
namespace isn't found on disk. The artifact is self-contained — clojure.string
works from any directory.
Drops the one-off nREPL source embed (jolt.nrepl is covered by this now). The
baked-in stdlib is excluded from uberscript bundles since it's part of the
runtime. clojure.core is unchanged (Janet, auto-referred).
Test: embedded-stdlib-test requires from the embedded copy with FS roots cleared.
Phase 3 — bundling. `jolt uberscript OUT -m NS` requires NS, uses the loader's
recorded load order (a dep finishes loading before its requirer, so it's
topological) to concatenate the used namespaces into one .clj that runs on a
plain jolt with no deps/jpm. `jolt-deps uberscript` resolves first. The loader
records loaded file paths when ctx env has :loaded-files.
Phase 4 — conformance. deps-conformance-test resolves real pure-cljc git libs
and reports load/run status, gated behind JOLT_CONFORMANCE so CI stays offline.
First run: medley works; cuerdas hits a reader gap (filed); stuartsierra/
dependency uses Long/MAX_VALUE (JVM interop, out of scope).
Tests: uberscript-test, deps-conformance-test (skips without the flag).
Add --version/version, -e/--eval, -f/--file, -m/--main (require NS and apply
its -main to *command-line-args*), and nrepl-server/--nrepl-server taking an
optional [host:]port (nrepl stays an alias). repl/help/-h round it out.
Also rewrite doc/tools-deps.md as design notes for the now-implemented deps
support (it still read as pre-implementation research), and note the new flags
in the README. Test: cli-test runs the flags from source.
Move build details, JOLT_PATH/namespace resolution, and the jolt-deps/deps.edn
usage into doc/building-and-deps.md instead of growing the README; README Build
now links to it.
A separate jolt-deps executable (mirroring jpm beside janet) resolves a
deps.edn into source roots and runs jolt with them on JOLT_PATH; the runtime
stays deps-agnostic.
- src/jolt/deps.janet: read deps.edn (Janet parses it directly), resolve
:git/* and :local/root via jpm/pm's download-bundle (git fetch + cache, no
build), recurse for transitive deps, return de-duped roots. jpm loaded
lazily so it's never embedded. Roots cached on a deps.edn hash.
- src/jolt/deps_cli.janet + project.janet: the jolt-deps tool — path/run/repl/-e.
- main: apply JOLT_PATH at runtime (the CLI ctx is frozen into the image at
build, so init's env read wouldn't see it).
Verified end to end against a local dep and a real git dep (medley). git only,
pure clj/cljc. Test: deps-resolve-test (local deps, no network).
The loader resolved a namespace against one hardcoded root (src/jolt, .clj
only). Now it searches an ordered list (ctx env :source-paths, stdlib first)
and tries .clj then .cljc. init adds roots from the :paths opt and JOLT_PATH.
This is the foundation for loading deps.edn libraries; verified loading medley
from an added root. No change to stdlib loading (3913 suite baseline held).
Drop Maven. Verified the piggyback: jpm's download-bundle clones a git dep
into its cache without building (the build half, bundle-install, needs a
project.janet and is separable). Plan is git resolution via jpm/pm, a loader
that searches the clones' src dirs, and compiling used namespaces into the
image at build. Pure clj/cljc only.
Findings and a phased plan for consuming deps.edn so projects can require
real Clojure libraries. Jolt already reads EDN, jars ship .clj/.cljc source,
and a real lib (medley) loads and runs. The gaps are a single-rooted loader
(needs a classpath + .cljc) and dependency resolution. Recommends reusing
'clojure -Spath' to start, then native git-dep resolution; jpm stays the
Janet build tool alongside this. See doc/tools-deps.md.
- grammar.ebnf: rewrite the number rule to cover the literal syntaxes the reader
now accepts — 0x/0X hex, N (bigint) / M (bigdec) suffixes, ratios a/b, radixed
integers (NrXXX, base 2..36), exponents, and the ##Inf/##-Inf/##NaN symbolic
floats — noting Jolt reads them as plain Janet numbers.
- README: Numbers bullet notes the literal syntaxes read; conformance count.
- reader-syntax-spec: drop the stale 'ratio not supported' case; add coverage
for hex-uppercase/N/M/ratio/radix/exponent/##Inf/##NaN.
- PLAN.md: refresh the stale Current State snapshot for the 3-layer test
structure (spec/integration/unit), ~3,920 suite assertions, 218/218
conformance, current source size.
jpm test green.
