Merge pull request #31 from jolt-lang/reader-feature-keys

Spec §2 (reader) + RFC 0002 feature-key decision + docs consolidation

docs/rfc/0002-reader-conditional-features.md

@@ -0,0 +1,82 @@
+# RFC 0002 — Reader-Conditional Feature Set
+
+- **Status**: Accepted (implemented; measured)
+- **Created**: 2026-06-10
+- **Spec**: `docs/spec/02-reader.md` §2.3 S18
+
+## Summary
+
+jolt's reader-conditional feature set is **`#{:jolt :default}`**, matched in
+**clause order** (the first clause whose key the platform satisfies wins).
+A loading context may opt a foreign, clj-targeted library into `:clj`
+compatibility via `reader-features-set!` (or process-wide via the
+`JOLT_FEATURES` environment variable). jolt does **not** satisfy `:clj` by
+default.
+
+## Background
+
+`#?(:clj … :cljs … :default …)` selects a branch by platform feature at read
+time. Until now jolt satisfied `:clj` — a compatibility shortcut inheriting
+the JVM branches of `.cljc` files, on the theory that the `:clj` branch is
+usually the "main" implementation. Each dialect chooses its own policy:
+ClojureScript satisfies only `:cljs`; jank uses `:jank`; babashka includes
+`:clj` because it genuinely is JVM-Clojure-compatible to a deep degree.
+
+Two defects forced the decision:
+
+1. jolt is *not* JVM-compatible where it matters for `:clj` branches: they
+   contain interop (`java.util.*`, `deftype` over JVM classes) and encode
+   JVM-specific *expectations* in tests (e.g. `parse-uuid`'s reference
+   permissiveness), both of which jolt fails.
+2. The old implementation also matched by **key priority** (`:clj` first,
+   then `:default`) rather than clause order — `#?(:default 5 :clj 6)` read
+   as `6`, diverging from Clojure on all platforms.
+
+## Decision and evidence
+
+Measured A/B over the cross-dialect clojure-test-suite (identical tree,
+2026-06-10):
+
+| Feature set | Assertions reached | Pass | Fail | Error | Clean files |
+|---|---|---|---|---|---|
+| `clj, default` (old) | 4967 | 4324 | 524 | 119 | 78 |
+| `jolt, default` (new) | **5069** | **4470** | **518** | **81** | **86** |
+
+The portable convention reads *more* of the suite (`:default` branches were
+being shadowed by `:clj` ones jolt can't satisfy) and improves every metric:
++146 passes, −38 errors, +8 clean files. The `:clj` shortcut was a net
+liability, not a compatibility win.
+
+The opposing case — loading real-world clj-targeted libraries — is real:
+SCI's `.cljc` sources select their implementation via `#?(:clj …)`/`:cljs`
+with no `:jolt` branches, and fail to load under the portable set. That is a
+property of the **loading context**, not of the platform: the resolution is
+per-context opt-in, exactly how the SCI bootstrap now loads
+(`(reader-features-set! ["jolt" "clj" "default"])`).
+
+## Specification (normative, mirrored in spec §2.3 S18)
+
+1. The platform feature set is implementation-defined and MUST be
+   documented. jolt's is `#{:jolt :default}`.
+2. Matching MUST be by clause order: the first clause whose key is in the
+   feature set wins. `:default` matches on every platform.
+   `#?(:default 5 :clj 6)` is `5` everywhere.
+3. An unmatched conditional reads as nothing (no form); an unmatched
+   `#?@(…)` splices nothing.
+4. Implementations SHOULD provide a per-loading-context override so foreign
+   libraries written for other dialects can be read under a compatibility
+   set; using it is a deliberate, scoped decision (jolt:
+   `reader-features-set!` / `JOLT_FEATURES`).
+
+## Consequences
+
+- Suite baselines re-measured and raised: `baseline-pass` 4324 → 4470,
+  `baseline-clean-files` 78 → 86.
+- Reader tests assert the portable set + clause-order semantics, plus one
+  opt-in round-trip through `reader-features-set!`.
+- Loading clj-ecosystem libraries via deps requires deciding their feature
+  set; the deps loader currently inherits the process default — a future
+  refinement is per-dependency feature configuration (filed with the deps
+  work, jolt-dw4).
+- `.cljc` authors targeting jolt can write `:jolt` branches and rely on
+  `:default` fallbacks.

docs/spec/00-front-matter.md

@@ -76,7 +76,7 @@ independent execution paths) and in the cross-dialect clojure-test-suite.
 | § | File | Status |
 |---|---|---|
 | 1 | `01-evaluation.md` | planned |
-| 2 | `02-reader.md` | planned (first: jank's reader corpus gives conformance shape) |
+| 2 | `02-reader.md` | **drafted** (grammar + reader-macro catalog; 2 divergences open) |
 | 3 | `03-special-forms.md` | **exemplars written** (`if`, `let*`); catalog complete |
 | 4 | `04-data-types.md` | planned (numeric-tower design note required) |
 | 5 | `05-sequences.md` | planned (laziness contract from jolt Phase-5 work) |

docs/spec/02-reader.md

@@ -0,0 +1,195 @@
+# §2 The Reader (Lexical Syntax)
+
+**Status**: token grammar drafted; reader-macro catalog complete with
+normative entries; syntax-quote section carries two open divergences.
+Conformance: jolt `reader-forms-spec` + `reader-syntax-spec` (granularity
+model: jank's per-construct corpus, 62 files under
+`test/jank/{reader-macro,syntax-quote}` — adapted rows cited per entry).
+
+The reader maps a stream of characters to *forms* (data). Reading is
+independent of evaluation: every form the reader produces is a value of the
+language (§4), and `read-string` exposes the reader as a function. Evaluation
+of forms is §1's concern; only `quote`-family reader macros reference it here.
+
+## 2.1 Tokens
+
+Whitespace is space, tab, newline, return, **and comma** (`,` is whitespace —
+S1). A `;` begins a comment to end of line (S2). Tokens:
+
+```
+form        := literal | symbol | keyword | list | vector | map | set
+             | reader-macro-form
+list        := '(' form* ')'
+vector      := '[' form* ']'
+map         := '{' (form form)* '}'
+literal     := nil | boolean | number | string | character
+nil         := 'nil'        boolean := 'true' | 'false'
+```
+
+- S3. A map literal MUST contain an even number of forms; duplicate keys
+  MUST be an error at read time.
+- S4. A set literal (`#{…}`, §2.3) with duplicate elements MUST be an error
+  at read time.
+
+### Numbers
+
+```
+integer  := ['+'|'-'] (digits | '0' [xX] hexdigits | '0' octdigits | radixR digits)
+float    := ['+'|'-'] digits '.' digits? exponent? | ['+'|'-'] digits exponent
+ratio    := ['+'|'-'] digits '/' digits            ; host-numeric-tower (§4 note)
+exponent := [eE] ['+'|'-'] digits
+```
+
+- S5. Trailing `N` (BigInt) and `M` (BigDecimal) suffixes are part of the
+  grammar; their value semantics are the §4 numeric-tower question.
+  Implementations without those towers SHOULD read them as the nearest
+  numeric type and MUST document the choice.
+
+### Symbols and keywords
+
+```
+symbol   := name | ns '/' name        ; '/' alone names the division fn
+keyword  := ':' name | ':' ns '/' name | '::' name | '::' alias '/' name
+```
+
+- S6. Symbol constituent characters: alphanumerics and `* + ! - _ ' ? < > =
+  . $ & %` (with `%` and `&` further constrained inside `#()`); a symbol
+  MUST NOT begin with a digit; `.` and `/` have positional restrictions.
+- S7. `::kw` MUST resolve to the current namespace at *read* time
+  (`::k` in ns `user` reads as `:user/k`); `::alias/k` resolves the alias or
+  MUST be a read error if the alias does not exist.
+
+### Strings and characters
+
+- S8. Strings are `"…"` with escapes `\" \\ \n \t \r \b \f \uNNNN \oNNN`.
+- S9. Character literals: `\c`, the named set `\newline \space \tab
+  \return \backspace \formfeed`, unicode `\uNNNN`, octal `\oNNN`.
+
+**Conformance** (2.1): jolt `reader-syntax-spec` "dispatch & sugar";
+clojure-test-suite reader files; jank `form/*` literal dirs. S3/S4 duplicate
+checks → UNVERIFIED (rows to add).
+
+## 2.2 Quote-family reader macros
+
+| Sugar | Reads as | |
+|---|---|---|
+| `'form` | `(quote form)` | S10 |
+| `@form` | `(deref form)` | S11 |
+| `^meta form` | form with metadata attached (see below) | S12 |
+| `#'sym` | `(var sym)` | S13 |
+| `` `form `` | syntax-quote (§2.4) | |
+| `~form`, `~@form` | unquote / unquote-splicing — only within syntax-quote (S14: MUST error outside) | |
+
+- S12a. `^:kw form` ≡ `^{:kw true} form`; `^Sym form` ≡ `^{:tag Sym} form`;
+  `^"str"` ≡ `^{:tag "str"} form`. Multiple `^` stack, rightmost innermost,
+  merged left-over-right.
+- S13a. `#'ns/sym` MUST denote the same var as `(var ns/sym)`:
+  `(= (var clojure.core/str) #'clojure.core/str)` is true.
+
+**Conformance**: jolt `reader-forms-spec` "var-quote #'", "metadata ^",
+"syntax-quote"; jank `var-quote/pass-qualified.jank`, `metadata/*`.
+
+## 2.3 Dispatch (`#`) reader macros
+
+| Form | Meaning | Entry |
+|---|---|---|
+| `#{…}` | set literal | S4 above |
+| `#"…"` | regex literal — reads to a regex value; escaping is regex-level, not string-level (single `\d`) | S15 |
+| `#(…)` | anonymous fn | S16 below |
+| `#_form` | discard | S17 below |
+| `#?(…)` / `#?@(…)` | reader conditional (+splicing) | S18 below |
+| `##Inf ##-Inf ##NaN` | symbolic floats | S19 |
+| `#tag form` | tagged literal | S20 below |
+| `#! …` | shebang comment line (implementations SHOULD accept) | |
+
+### S16 — anonymous function `#(…)`
+
+- `#(body)` reads as `(fn [args…] (body))` with parameters derived from the
+  `%`-symbols appearing in body: `%`≡`%1`, `%n` positional, `%&` the rest
+  parameter. Arity = highest `%n` mentioned (plus rest if `%&`).
+- `#()` literals MUST NOT nest.
+
+```clojure
+(#(+ %1 %2) 1 2)            ;=> 3
+(apply #(apply + %&) [1 2 3]) ;=> 6
+(map #(* % %) [1 2])        ;=> (1 4)
+```
+
+### S17 — discard `#_`
+
+- `#_form` reads and discards the next form entirely (it is never evaluated).
+- Discards compose: `#_ #_ a b` discards two following forms.
+- `#_` inside collection literals removes the element: `[1 #_2 3]` ⇒ `[1 3]`.
+
+### S18 — reader conditionals
+
+- `#?(:feat₁ f₁ :feat₂ f₂ …)` reads as the form of the first feature key the
+  platform satisfies, else nothing. `:default` matches any platform.
+  `#?@(…)` splices a sequential form into the surrounding context.
+- Feature keys are implementation-defined; each implementation MUST document
+  its feature set, and SHOULD follow the portable convention *own dialect key
+  + `:default`*. Matching MUST be by **clause order** — the first clause whose
+  key the platform satisfies wins (`#?(:default 5 :clj 6)` is `5` everywhere)
+  — not by key priority. Implementations SHOULD provide a per-loading-context
+  compatibility override for foreign-dialect libraries. (jolt:
+  `#{:jolt :default}`, opt-in via `reader-features-set!`/`JOLT_FEATURES`;
+  decision + A/B data in RFC 0002 — inheriting `:clj` cost 146 suite
+  assertions and 38 errors.)
+- Reader conditionals MUST be an error outside `.cljc`-style reading unless
+  the implementation documents otherwise.
+
+### S19 — symbolic values
+
+`##Inf`, `##-Inf`, `##NaN` read as the IEEE-754 values. `(= ##NaN ##NaN)` is
+false; `(NaN? ##NaN)` is true.
+
+### S20 — tagged literals
+
+- `#tag form`: the reader resolves `tag` in the data-reader table and MUST
+  apply the reader function to the *read* form, yielding its result as the
+  read value. An unknown tag MUST be a read error (jank
+  `fail-unsupported-tag`).
+- Built-in tags every implementation MUST provide: `#uuid "…"` → a UUID
+  value (§9 `parse-uuid` semantics — round-trips through printing), `#inst
+  "…"` → an instant (representation host-classified; ⚠ jolt currently reads
+  `#inst` as its bare string — divergence, filed).
+
+**Conformance** (2.3): jolt `reader-forms-spec` "#() (% %N %&)" + new rows
+(symbolic values, stacked discard, conditionals); `uuid-spec` reader-literal
+group; jank `reader-macro/{function,regex,uuid,symbolic-value}/*`,
+`fail-unsupported-tag.jank`.
+
+## 2.4 Syntax-quote
+
+Syntax-quote (`` ` ``) is read-level template construction with namespace
+resolution:
+
+- S21. Inside syntax-quote, an unqualified symbol that resolves in
+  `clojure.core` MUST be qualified to `clojure.core/sym`; a symbol resolving
+  through a namespace alias MUST be qualified to the aliased namespace; an
+  unresolved symbol MUST be qualified to the current namespace. Special-form
+  names stay bare.
+- S22. `sym#` generates a fresh symbol, stable *within one syntax-quote
+  template* (all `sym#` in the same template denote the same generated
+  symbol; distinct templates generate distinct symbols).
+- S23. `~form` inserts the value of `form`; `~@form` splices a sequential
+  value; `~'sym` is the idiom for an intentionally-unqualified symbol.
+- S24. Syntax-quote distributes through collection literals (vectors, maps,
+  sets) — qualification and unquoting apply inside them.
+- S25. Nested syntax-quotes expand recursively (quasiquote semantics); a
+  syntax-quoted self-evaluating literal is the literal: `(= "meow"
+  ```"meow")` is true. ⚠ jolt currently fails S25 (filed) — UNVERIFIED
+  pending the fix.
+
+**Conformance**: jolt `reader-forms-spec` "syntax-quote" (gensym, unquote,
+splice) + conformance "syntax-quote fully-qualifies"; jank
+`syntax-quote/{pass-gensym,pass-namespace-resolution,pass-resolve-alias,
+unquote,unquote-splice}/*`. S25 → UNVERIFIED.
+
+## 2.5 What the reader is not
+
+The reader performs **no macroexpansion and no evaluation** (tagged-literal
+reader functions are the deliberate exception, S20). Forms read identically
+whether or not they will be evaluated; `read-string` of any printable value
+`v` followed by evaluation yields a value equal to `v` for the
+self-evaluating types (§4 print/read round-trip contract).

src/jolt/reader.janet

@@ -387,6 +387,34 @@
       (array/push arg-names rest-sym))
     [@[(sym "fn*") (tuple ;arg-names) replaced] new-pos]))
 
+# The reader-conditional feature set (spec 02-reader S18): jolt is its own
+# dialect, so the portable convention applies — the dialect key plus :default.
+# Matching is by CLAUSE order (the first clause whose key is in the feature
+# set wins), exactly like Clojure — NOT key-priority. JOLT_FEATURES overrides
+# (comma-separated, e.g. "jolt,clj,default") for compat experiments and A/B
+# measurement; :default is always honored.
+# Mutable so a loading context can opt a clj-targeted foreign library into
+# :clj compatibility (e.g. SCI) — see reader-features-set!. jolt itself and
+# the conformance surface read under the portable set.
+(var reader-features nil)
+
+(defn reader-features-set!
+  "Replace the active reader-conditional feature set (a list of keyword-name
+  strings or keywords). :default is always honored. Returns the previous set
+  so callers can restore."
+  [names]
+  (def prev reader-features)
+  (def t @{})
+  (each n names (put t (if (keyword? n) n (keyword n)) true))
+  (put t :default true)
+  (set reader-features t)
+  prev)
+
+(reader-features-set!
+  (let [env (os/getenv "JOLT_FEATURES")]
+    (if env (filter |(> (length $) 0) (string/split "," env))
+      ["jolt" "default"])))
+
 (defn read-reader-conditional [s pos]
   # pos is at #, next char is ? or ?@
   (def splice? (and (< (+ pos 2) (length s)) (= (s (+ pos 2)) 64))) # @ = 64
@@ -394,23 +422,16 @@
   (let [[form new-pos] (read-form s form-start)]
     (if (array? form)
       (do
+        # First clause (in clause order) whose feature key is in the set.
+        # `matched` is tracked separately: a matched clause may be nil.
         (var result nil)
+        (var matched false)
         (var i 0)
-        (while (< i (length form))
+        (while (and (not matched) (< i (length form)))
-          (if (= (in form i) :clj)
+          (when (get reader-features (in form i))
-            (do
+            (set result (in form (+ i 1)))
-              (set result (in form (+ i 1)))
+            (set matched true))
-              (set i (length form)))
+          (+= i 2))
-            (++ i)))
-        # Fallback to :default if :clj not matched
-        (when (nil? result)
-          (set i 0)
-          (while (< i (length form))
-            (if (= (in form i) :default)
-              (do
-                (set result (in form (+ i 1)))
-                (set i (length form)))
-              (++ i))))
         (if splice?
           # #?@ splicing: resolve :clj branch, wrap for splice
           (let [items (if (nil? result)

test/integration/clojure-test-suite-test.janet

@@ -43,9 +43,9 @@
 # Raised 4004 -> 4034 / clean 66 -> 67 porting partition-all + repeatedly to the
 # overlay, which required fixing two leniencies (a char is not callable; take
 # validates its count) — correct beyond those fns, so the suite rose broadly.
-(def baseline-pass 4324)
+(def baseline-pass 4470)
 # A file is "clean" when it ran with zero failures AND zero errors.
-(def baseline-clean-files 78)
+(def baseline-clean-files 86)
 # Per-file wall-clock budget (seconds). Normal files finish in well under 1s, so
 # this normally only fires on genuinely-infinite-sequence hangs. It's an env var
 # (JOLT_SUITE_TIMEOUT) so CI — whose runners are slower than a dev machine — can

test/integration/sci-bootstrap-test.janet

@@ -2,6 +2,12 @@
 (use ../../src/jolt/types)
 (use ../../src/jolt/reader)
 (use ../../src/jolt/api)
+(use ../../src/jolt/reader)
+# SCI is a clj/cljs-targeted library: its .cljc sources select implementation
+# via #?(:clj ...) and have no :jolt branches — load it under clj-compat
+# features (spec 02-reader S18: feature sets are a property of the loading
+# context; the portable default is #{:jolt :default}).
+(reader-features-set! ["jolt" "clj" "default"])
 
 (def ctx (init))
 

test/integration/sci-runtime-test.janet

@@ -2,6 +2,12 @@
 (use ../../src/jolt/types)
 (use ../../src/jolt/reader)
 (use ../../src/jolt/api)
+(use ../../src/jolt/reader)
+# SCI is a clj/cljs-targeted library: its .cljc sources select implementation
+# via #?(:clj ...) and have no :jolt branches — load it under clj-compat
+# features (spec 02-reader S18: feature sets are a property of the loading
+# context; the portable default is #{:jolt :default}).
+(reader-features-set! ["jolt" "clj" "default"])
 
 (defn- load-stubs [ctx filepath]
   (var s (slurp filepath))

test/spec/reader-forms-spec.janet

@@ -48,3 +48,18 @@
   # jolt-edb (fixed): ~/~@ inside set literals.
   ["splice in set"   "#{0 1 2 3}" "(let [b [0] a [1 2 3] e []] `#{~@e ~@a ~@b})"]
   ["unquote in set"  "#{5 9}"   "(let [x 5] `#{~x 9})"])
+
+# Spec 02-reader S17/S19/S18/S13a/S22: discard, symbolic values, conditionals,
+# var-quote identity, gensym stability (jank corpus derived).
+(defspec "reader / discard, symbolic values, conditionals (spec 2.3)"
+  ["discard simple"      "2"     "(do #_1 2)"]
+  ["discard in vector"   "[1 3]" "[1 #_2 3]"]
+  ["discard stacks"      "3"     "(do #_ #_ 1 2 3)"]
+  ["##Inf"               "true"  "(= ##Inf (/ 1.0 0.0))"]
+  ["##-Inf"              "true"  "(= ##-Inf (/ -1.0 0.0))"]
+  ["##NaN not self-equal" "false" "(= ##NaN ##NaN)"]
+  ["##NaN is NaN?"       "true"  "(NaN? ##NaN)"]
+  ["conditional :default reachable" "2" "#?(:no-such-dialect 1 :default 2)"]
+  ["var-quote qualified" "true"  "(= (var clojure.core/str) #'clojure.core/str)"]
+  ["gensym stable in template" "true" "(let [syms `[meow# meow#]] (= (first syms) (second syms)))"]
+  ["gensym fresh across templates" "false" "(= `meow# `meow#)"])

test/spec/reader-syntax-spec.janet

@@ -45,8 +45,14 @@
   ["anon fn %&"         "[2 3]"     "(#(vec %&) 2 3)"]
   ["discard #_"         "[1 3]"     "[1 #_2 3]"]
   ["regex literal"      "true"      "(= \"abc\" (re-find #\"abc\" \"xabcx\"))"]
-  ["reader conditional" "1"         "#?(:clj 1 :cljs 2 :default 3)"]
+  # Feature set is #{:jolt :default} (spec 02-reader S18; RFC 0002) — :clj
-  ["reader cond splice" "[1 2 3]"   "[#?@(:clj [1 2 3] :cljs [4 5])]"]
+  # branches are NOT taken; matching is by clause order.
+  ["reader conditional" "3"         "#?(:clj 1 :cljs 2 :default 3)"]
+  ["reader cond :jolt"  "4"         "#?(:clj 1 :jolt 4 :default 3)"]
+  ["reader cond clause order" "5"   "#?(:default 5 :jolt 6)"]
+  ["reader cond no match" "[]"      "[#?(:clj 1 :cljs 2)]"]
+  ["reader cond splice" "[1 2 3]"   "[#?@(:jolt [1 2 3] :cljs [4 5])]"]
+  ["reader cond splice no match" "[]" "[#?@(:clj [1 2 3] :cljs [4 5])]"]
   ["inst literal reads" "true"      "(some? #inst \"2020-01-01T00:00:00Z\")"]
   ["uuid literal"       "\"550e8400-e29b-41d4-a716-446655440000\"" "(str #uuid \"550e8400-e29b-41d4-a716-446655440000\")"]
   ["tagged literal var" "true"      "(var? #'+)"]

test/unit/reader-test.janet

@@ -123,18 +123,23 @@
   (let [[form2 _] (parse-next rest-str)]
     (assert (deep= [3 4] form2) "second form is vector")))
 
-# Reader conditional — resolves :clj branch at read time
+# Reader conditional — feature set is #{:jolt :default} (spec 02-reader S18,
-(assert (= 1 (parse-string "#?(:clj 1 :cljs 2)"))
+# RFC 0002); matching is by clause order. reader-features-set! lets a loading
-        "#?(:clj) picks :clj branch")
+# context opt a clj-targeted library into :clj compat (restored below).
+(assert (= 3 (parse-string "#?(:clj 1 :jolt 3 :cljs 2)"))
+        "#?(...) picks :jolt branch")
 (assert (= nil (parse-string "#?(:cljs 999)"))
-        "#?(:cljs) returns nil on CLJ")
+        "unmatched conditional reads as nothing")
-(assert (= 42 (parse-string "#?(:clj 42)"))
+(assert (= 7 (parse-string "#?(:clj 1 :default 7)"))
-        "#?(:clj) with single branch")
+        ":default reached when :clj not in feature set")
-(assert (deep= (sym "clj-only") (parse-string "#?(:clj clj-only :cljs cljs-only)"))
+(assert (= 5 (parse-string "#?(:default 5 :jolt 6)"))
-        "#?(:clj) picks :clj symbol")
+        "clause order wins, not key priority")
-# Nested inside a list — :clj branch is evaluated at read time
+(assert (deep= @[(sym "+") 1 3] (parse-string "(+ 1 #?(:jolt 3 :cljs 4))"))
-(assert (deep= @[(sym "+") 1 3] (parse-string "(+ 1 #?(:clj 3 :cljs 4))"))
+        "#? inside list picks :jolt")
-        "#? inside list picks :clj")
+(let [prev (reader-features-set! ["jolt" "clj" "default"])]
+  (assert (= 1 (parse-string "#?(:clj 1 :cljs 2)"))
+          "clj-compat opt-in picks :clj")
+  (reader-features-set! (keys prev)))
 
 # Characters — the reader now produces char values {:jolt/type :jolt/char :ch N}
 (let [form (parse-string "\\newline")]