534007641e
* core: compile macro expanders via staged bootstrap (Stage 2 Task 1)
Macros are now compiled, not interpreted, by steady state — matching
Clojure (macros are ordinary compiled fns the Java seed compiles for
clojure.core) and ClojureScript (macros compiled, invoked at compile
time). Neither reference keeps an interpreted-closure fallback.
The early macros (00-syntax) are defined while the self-hosted analyzer
is still being bootstrapped (it builds lazily after the overlay loads),
so macro-compile-hook returns nil and they get an interpreted closure.
The bootstrap compiler.janet can't substitute (it punts on syntax-quote,
which nearly every expander uses).
Fix = staged bootstrap, the same pattern as the compiler fixpoint:
defmacro stashes the expander source on the var (:macro-src) plus a
:macro-uses-env flag; once the overlay + analyzer are fully built,
backend/recompile-macros! (via ensure-macros-compiled! at the end of
load-core-overlay!) compiles each stashed expander through the now-live
analyzer and rebinds the var, marking :macro-compiled. Idempotent;
&env/&form macros keep the interpreted closure (the compiled fn* has no
such params). The interpreter is now a build-time crutch, gone by
steady state.
Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring
(which the analyzer punts on) to a plain rest param + (first rest), so
all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79.
47/47 overlay macros compiled, 0 interpreted; user macros compile
immediately post-init. Gate green: conformance 267x3, fallback-zero
31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap,
lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap,
features 78/78, all unit+spec, core-bench neutral.
* core: wrap macro expanders in fn (not fn*) so destructured arglists compile
Follow-up to the staged macro-compile change. The macro-recompile pass
wrapped each expander in the raw fn* primitive, which punts on a
destructuring rest param — so if-not/if-let/if-some/assert (using the
canonical '& [else]') couldn't compile and had been rewritten to plain
rest params as a workaround.
Root cause: fn* is the primitive; the fn MACRO is what desugars
destructuring (rest, map, nested) into the body before lowering. Wrapping
expanders in fn instead of fn* compiles any destructured macro arglist
uniformly, so the workaround is unnecessary — reverted those 4 macros to
the canonical '& [else]' forms.
Net: rest-destructuring is fully compiled for all normal code (fn/defn/
let/macro params). Only the hand-written raw fn* primitive still punts
(jolt-f79, downgraded to P4 — falls back to interpreter, still correct).
47/47 overlay macros compiled. Gate green: conformance 267x3,
fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host,
staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67,
sci-bootstrap, features, all unit+spec.
* core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79)
jolt-f79 asked to compile destructuring in fn* rest params. Checking
against Clojure inverts the premise: Clojure's fn* REJECTS destructuring
at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding
form, expected symbol'). So the self-hosted analyzer was already correct
— fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn
MACROS desugar destructuring. The real defect was the interpreter
leniently destructuring raw fn*, and defn emitting raw fn* to rely on it.
Changes:
- evaluator: fn*/let*/loop* now reject non-symbol binding forms with
Clojure's exact messages (require-symbol-params/plain-sym?), so the
interpreter agrees with the analyzer + Clojure.
- 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring
params desugar; unnamed, so self-recursion still resolves via the var.
- 00-syntax: completing that exposed a real gap — the overlay destructure
fn didn't handle kwargs (a map pattern bound against a fn's sequential
rest); it had only worked via the interpreter's destructure-bind. Added
the seq->map coercion to the map? branch (sequential: 1 map elt => that
map, else apply hash-map), matching destructure-bind so interpret ==
compile.
Net: fn*/let*/loop* are plain-symbol primitives across interpreter,
analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro
params, incl kwargs & {:keys}) compiles through the macros with no
interpreter fallback. Regression spec added.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44,
clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all
unit+spec (destructuring 50/50).
* docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79)
grammar.ebnf: the destructuring note already attributed patterns to the
binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES
they desugar to take plain symbols only (a non-symbol binding errors, as
in Clojure).
self-hosting-compiler.md: the 'compile destructuring via a shared
destructure expander instead of falling back' item is done (and its
jolt-7dl ref was stale) — destructuring now compiles through the
fn/let/loop/defn macros' desugaring; the primitives reject patterns.
* core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy
First slice of moving stateful forms onto the compile path (jolt-eaa).
- loader stateful-head?: drop syntax-quote (the analyzer's `handled` set
already compiles it; routing it to the interpreter was redundant).
- host_iface special-names: drop definterface/extend/proxy (stub macros
expanding to def/nil — their expansions compile once unpunted).
- letfn stays interpreted: its let* expansion needs letrec semantics
(mutual recursion between the fns), which sequential compiled let* lacks
— a later tier.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + protocol/multimethod/macro specs.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
185 lines
9.2 KiB
EBNF
185 lines
9.2 KiB
EBNF
(* ===========================================================================
|
|
Jolt reader grammar (EBNF)
|
|
===========================================================================
|
|
|
|
This grammar specifies the surface syntax accepted by Jolt's reader
|
|
(src/jolt/reader.janet) — the text that `read`/`parse-string`/`load-string`
|
|
turn into data/forms. It is the syntactic half of Jolt's contract; the
|
|
behavioural half lives in test/spec/. Where Jolt diverges from Clojure the
|
|
difference is called out in a comment.
|
|
|
|
Notation (ISO-ish EBNF):
|
|
= definition | alternation
|
|
, concatenation ( ) grouping
|
|
[ x ] optional (0 or 1) { x } repetition (0 or more)
|
|
"x" 'x' terminal (* *) comment
|
|
? x ? informal/char-class terminal
|
|
|
|
A source file is a sequence of forms; evaluation is form-by-form.
|
|
=========================================================================== *)
|
|
|
|
file = { ws | form } ;
|
|
|
|
(* --------------------------------------------------------------------------
|
|
Whitespace, separators and comments
|
|
-------------------------------------------------------------------------- *)
|
|
|
|
ws = { whitespace-char | comment | discard } ;
|
|
whitespace-char = " " | "\t" | "\n" | "\r" | "," ; (* comma is whitespace *)
|
|
comment = ";" , { ? any char except "\n" ? } , [ "\n" ] ;
|
|
discard = "#_" , ws , form ; (* read then drop next form *)
|
|
|
|
(* --------------------------------------------------------------------------
|
|
Forms
|
|
-------------------------------------------------------------------------- *)
|
|
|
|
form = scalar
|
|
| collection
|
|
| reader-macro
|
|
| dispatch ;
|
|
|
|
scalar = nil | boolean | number | string | character
|
|
| keyword | symbol ;
|
|
|
|
collection = list | vector | map ;
|
|
|
|
(* --------------------------------------------------------------------------
|
|
Atoms / scalars
|
|
-------------------------------------------------------------------------- *)
|
|
|
|
nil = "nil" ;
|
|
boolean = "true" | "false" ;
|
|
|
|
(* Numbers. Jolt accepts Clojure's numeric literal syntaxes, but — since Jolt
|
|
numbers are Janet ints/doubles — it has no distinct bignum, ratio or
|
|
BigDecimal types: the BigInt suffix N and BigDecimal suffix M are read as the
|
|
plain number, a ratio a/b is read as its double quotient, and radixed
|
|
integers are computed by base. The symbolic floats ##Inf/##-Inf/##NaN are
|
|
also read. (No octal-with-leading-0 literal.) *)
|
|
number = symbolic-value
|
|
| [ sign ] , ( radix-int | ratio | hex-int | decimal ) ;
|
|
sign = "+" | "-" ;
|
|
integer = digit , { digit } ;
|
|
hex-int = "0" , ( "x" | "X" ) , hex-digit , { hex-digit } , [ "N" ] ;
|
|
radix-int = integer , ( "r" | "R" ) , alnum , { alnum } ; (* base 2..36: 2r1010, 16rFF, 36rZ *)
|
|
ratio = integer , "/" , integer ; (* read as a double quotient *)
|
|
decimal = integer , [ "." , digit , { digit } ] , [ exponent ] , [ num-suffix ] ;
|
|
exponent = ( "e" | "E" ) , [ sign ] , digit , { digit } ;
|
|
num-suffix = "N" | "M" ; (* BigInt / BigDecimal in Clojure; plain number in Jolt *)
|
|
symbolic-value = "##Inf" | "##-Inf" | "##NaN" ;
|
|
digit = "0".."9" ;
|
|
hex-digit = digit | "a".."f" | "A".."F" ;
|
|
alnum = digit | "a".."z" | "A".."Z" ;
|
|
|
|
(* Strings: double-quoted with backslash escapes. *)
|
|
string = '"' , { string-char } , '"' ;
|
|
string-char = ? any char except '"' and "\" ? | escape ;
|
|
escape = "\" , ( "n" | "t" | "r" | "\" | '"' | ? other ? ) ;
|
|
|
|
(* Characters: \x, a named char, or \uNNNN / \oNNN. *)
|
|
character = "\" , ( char-name | ? any single char ? ) ;
|
|
char-name = "newline" | "space" | "tab" | "return"
|
|
| "formfeed" | "backspace" | "nul"
|
|
| "u" , hex-digit , hex-digit , hex-digit , hex-digit
|
|
| "o" , octal-digit , { octal-digit } ;
|
|
octal-digit = "0".."7" ;
|
|
|
|
(* Keywords: :name, :ns/name, or auto-resolved ::name. *)
|
|
keyword = "::" , sym-token (* auto-resolved to current ns *)
|
|
| ":" , sym-token ;
|
|
|
|
(* Symbols: a token of symbol-constituent chars not parsed as a number. A "/"
|
|
separates an optional namespace from the name. *)
|
|
symbol = sym-token ;
|
|
sym-token = sym-start , { sym-constituent } ;
|
|
sym-start = sym-constituent - ( digit ) ; (* not a leading digit *)
|
|
sym-constituent = letter | digit
|
|
| "*" | "+" | "!" | "_" | "-" | "?" | "." | "<" | ">"
|
|
| "=" | "&" | "|" | "$" | "%" | "/" ;
|
|
letter = "a".."z" | "A".."Z" ;
|
|
|
|
(* --------------------------------------------------------------------------
|
|
Collections
|
|
-------------------------------------------------------------------------- *)
|
|
|
|
list = "(" , { ws | form } , ")" ; (* -> a Clojure list *)
|
|
vector = "[" , { ws | form } , "]" ; (* -> a persistent vector *)
|
|
map = "{" , { ws | map-entry } , "}" ;
|
|
map-entry = form , ws , form ; (* an even number of forms *)
|
|
|
|
(* --------------------------------------------------------------------------
|
|
Reader macros (prefix sugar). Each expands to a 2-element form
|
|
(op operand), e.g. 'x -> (quote x), @a -> (deref a).
|
|
-------------------------------------------------------------------------- *)
|
|
|
|
reader-macro = quote | syntax-quote | unquote | unquote-splice
|
|
| deref | metadata ;
|
|
|
|
quote = "'" , form ; (* (quote form) *)
|
|
syntax-quote = "`" , form ; (* (syntax-quote form) *)
|
|
unquote-splice = "~@" , form ; (* (unquote-splicing form) *)
|
|
unquote = "~" , form ; (* (unquote form) *)
|
|
deref = "@" , form ; (* (deref form) *)
|
|
metadata = "^" , meta-form , ws , form ; (* attach metadata to form *)
|
|
meta-form = map | keyword | symbol | string ;
|
|
(* Normalized like Clojure: a symbol or string is a type hint -> {:tag ...};
|
|
a keyword -> {keyword true}; a map is used as-is. On a symbol the metadata
|
|
rides on the symbol (it stays a bare symbol, so a hint like ^String is
|
|
transparent in params/lets/bodies); other targets use a runtime with-meta. *)
|
|
|
|
(* --------------------------------------------------------------------------
|
|
Dispatch forms — introduced by "#".
|
|
-------------------------------------------------------------------------- *)
|
|
|
|
dispatch = set
|
|
| anon-fn
|
|
| var-quote
|
|
| regex
|
|
| reader-conditional
|
|
| discard (* #_form — see Whitespace above *)
|
|
| tagged-literal ;
|
|
|
|
set = "#{" , { ws | form } , "}" ; (* a persistent set *)
|
|
|
|
(* Anonymous function. %, %1.. are positional params; %& is the rest param. *)
|
|
anon-fn = "#(" , { ws | form } , ")" ; (* -> (fn* [..] body) *)
|
|
anon-arg = "%" | "%" , digit , { digit } | "%&" ;
|
|
|
|
var-quote = "#'" , symbol ; (* (var symbol) *)
|
|
|
|
(* Regex literal -> a Janet PEG-backed regex value.
|
|
Supported: groups, greedy/lazy quantifiers, (?:..), lookahead (?=..)/(?!..),
|
|
alternation, anchors ^ $ \b \B, classes, (?i). NOT: lookbehind,
|
|
backreferences, named groups. *)
|
|
regex = '#"' , { ? any char except '"' ? | "\" , ? any char ? } , '"' ;
|
|
|
|
(* Reader conditional. Jolt reads as the :clj platform; #?@ splices a sequence. *)
|
|
reader-conditional
|
|
= "#?@" , "(" , { ws | feature , ws , form } , ")"
|
|
| "#?" , "(" , { ws | feature , ws , form } , ")" ;
|
|
feature = ":clj" | ":cljs" | ":default" | keyword ;
|
|
|
|
(* Tagged literal: #tag form. Built-in tags include #inst and #uuid; others
|
|
dispatch to a registered data-reader (else error). *)
|
|
tagged-literal = "#" , symbol , ws , form ;
|
|
|
|
(* ── Destructuring (semantics, not reader syntax) ────────────────────────────
|
|
The reader produces plain vectors and maps; the binding MACROS (let, fn, loop,
|
|
doseq, for, defmacro params, …) interpret them as destructuring patterns. The
|
|
PRIMITIVES they desugar to — fn*, let*, loop* — take plain symbols ONLY (a
|
|
non-symbol binding is an error, as in Clojure: "fn params must be Symbols" /
|
|
"Bad binding form, expected symbol"). The grammar of a pattern:
|
|
|
|
binding = symbol | seq-binding | map-binding ;
|
|
seq-binding = "[" , { binding } , [ "&" , binding ] , [ ":as" , symbol ] , "]" ;
|
|
map-binding = "{" , { binding , form (* {local key} *)
|
|
| ":keys" , "[" {symbol} "]" (* x or ns/x *)
|
|
| ":strs" , "[" {symbol} "]"
|
|
| ":syms" , "[" {symbol} "]"
|
|
| ":or" , map (* defaults *)
|
|
| ":as" , symbol } , "}" ;
|
|
|
|
A symbol in :keys/:syms may be namespaced (x/y): the namespaced key is looked
|
|
up but the local is the bare name (y). Destructuring a *sequence* with a
|
|
map-binding treats it as keyword args (alternating k v, or a trailing map) —
|
|
this is the fn/macro `[& {:keys [...]}]` form. *)
|