Route syntax-quote, set!, var, ., new to interpreter

Compile-time implementation of syntax-quote proved too complex for the
current emission pipeline (qualified symbol handling across both
string and data-structure paths). These ops require runtime context
(var lookup, field mutation, deftype construction) and are now correctly
routed to the interpreter via the stateful? check in eval-string.

Also fixed:
- raw-form->janet handles namespace-qualified symbols correctly
- emit-quote-str and emit-quote-expr use raw-form->janet consistently
- Duplicate function definitions removed

All 317 tests pass, 0 failures.
This commit is contained in:
Yogthos 2026-06-02 16:48:10 -04:00
parent 1de109f261
commit 4ca7c31e50
7 changed files with 254 additions and 44 deletions

View file

@ -1,7 +1,37 @@
Janet's `eval` runs in Janet's default environment and does NOT have access to symbols imported via `(use ...)` in the calling file. `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. FIX: emit Janet data structures where function VALUES are embedded directly (e.g. `[core-inc 1]`) rather than source strings `"(core-inc 1)"`. The `core-fn-values` table resolves Janet symbol names to actual function values at compile time.
§
Jolt Compiler Architecture (Phases 1-6, dfa9874→c366963): Two-phase — analyze-form (Clojure form → annotated AST) → emit-ast (→ Janet source string) or emit-expr (→ Janet data structures for eval). analyze-form takes [form bindings &opt ctx]; ctx needed for macro expansion. Symbol classification: bindings first (:local), then core-renames (:core-symbol), then plain (:symbol). Two emitter paths: string (compile-form) and data structures (compile-ast). Core fn values resolved via core-fn-values table. compile-and-eval takes [form ctx]; pass nil for no macro ctx. Naming: Clojure - → core-sub. Key bugs found: emit-vector-expr must wrap with (tuple ...) — bare tuples eval as fn calls. make-symbol must treat / at pos 0 as unqualified. raw-form->janet converter for quote (don't re-analyze). core-renames entry for fn? was missing.
Jolt Compiler Architecture (Phases 1-6, dfa9874→1de109f): Two-phase — analyze-form (Clojure form → annotated AST) → emit-ast (→ Janet source string) or emit-expr (→ Janet data structures for eval). analyze-form takes [form bindings &opt ctx]; ctx needed for macro expansion. Symbol classification: bindings first (:local), then core-renames (:core-symbol), then plain (:symbol). Two emitter paths: string (compile-form) and data structures (compile-ast). Core fn values resolved via core-fn-values table. compile-and-eval takes [form ctx]; pass nil for no macro ctx.
Key naming/facts:
- Clojure - → core-sub (NOT core--)
- core-nth did not exist — had to add both the function and core-bindings entry
- Missing from core-renames early: fn?, list, name, subs
- Bare tuples in Janet eval → treated as function calls. Always emit (tuple ...) or ['tuple ...]
- make-symbol: / at position 0 means unqualified symbol (was parsing empty ns)
- raw-form->janet converter for quote: don't re-analyze quoted forms, pass through verbatim
- emit-try-expr: Janet format is (try body ([err] handler)) not (try body (catch sym handler))
- Loop compilation: (do (var _loop_N nil) (set _loop_N (fn [params] body)) (_loop_N init-vals...))
- Recur compilation: rewrites to (loop-name arg1 arg2...) via :loop-name in AST
eval-string dispatch: When :compile? true, stateful forms (defmacro, ns, deftype, defmulti, defmethod, require, in-ns) use interpreter. All others (def, macros like defn) go through compile-and-eval. Macros expanded at analyze time via resolve-macro.
Remaining: syntax-quote, set! compiler support. deftype/defmulti/defmethod routed to interpreter.
§
eval-string dispatch (api.janet): When :compile? is true, stateful forms (defmacro, ns, deftype, defmulti, defmethod, require, in-ns) always use interpreter. All others (def, defn via macros, pure expressions) go through compile-and-eval. defn bug was caused by macros not being expanded — fixed in Phase 4 by routing macros through compiler's analyze-form which expands them at analyze time.
Jolt Compiler Architecture (Phases 1-6, dfa9874→1de109f): Two-phase — analyze-form (Clojure form → annotated AST) → emit-ast (→ Janet source string) or emit-expr (→ Janet data structures for eval). analyze-form takes [form bindings &opt ctx]; ctx needed for macro expansion. Symbol classification: bindings first (:local), then core-renames (:core-symbol), then plain (:symbol). Two emitter paths: string (compile-form) and data structures (compile-ast). Core fn values resolved via core-fn-values table. compile-and-eval takes [form ctx]; pass nil for no macro ctx.
Key naming/facts:
- Clojure - → core-sub (NOT core--)
- core-nth did not exist — had to add both the function and core-bindings entry
- Missing from core-renames early: fn?, list, name, subs
- Bare tuples in Janet eval → treated as function calls. Always emit (tuple ...) or ['tuple ...]
- make-symbol: / at position 0 means unqualified symbol (was parsing empty ns)
- raw-form->janet converter for quote: don't re-analyze quoted forms, pass through verbatim
- emit-try-expr: Janet format is (try body ([err] handler)) not (try body (catch sym handler))
- Loop compilation: (do (var _loop_N nil) (set _loop_N (fn [params] body)) (_loop_N init-vals...))
- Recur compilation: rewrites to (loop-name arg1 arg2...) via :loop-name in AST
eval-string dispatch: When :compile? true, stateful forms (defmacro, ns, deftype, defmulti, defmethod, require, in-ns) use interpreter. All others (def, macros like defn) go through compile-and-eval. Macros expanded at analyze time via resolve-macro.
Remaining: syntax-quote, set! compiler support. deftype/defmulti/defmethod routed to interpreter. Git push needs manual approval.
§
Phase 6: 47 comprehensive compile-mode tests in test/phase6-final.janet. Collections, math, predicates, comparison, seq ops (map/filter/reduce/take/drop), special forms (let/if/loop/try/quote), macros (defn/when/and/or/fn/if-let), complex nesting. All 317 tests pass. Remaining: syntax-quote, set! compiler support. deftype, defmulti/defmethod routed to interpreter (stateful).
Phase 6: 47 comprehensive compile-mode tests in test/phase6-final.janet. Collections, math, predicates, comparison, seq ops (map/filter/reduce/take/drop), special forms (let/if/loop/try/quote), macros (defn/when/and/or/fn/if-let), complex nesting. 58 assertions. All 317 tests pass, 0 failures. Remaining: syntax-quote, set! compiler support. deftype, defmulti/defmethod routed to interpreter (stateful).

View file

@ -1,5 +1,9 @@
Janet eval data-structure constraints: (1) Bare tuples eval as fn calls — use `['tuple val1 val2]` not `[val1 val2]` for vector values. (2) Janet try: `[(tuple ;[err-sym]) handler-body]` NOT `(catch [err] body)` — catch is NOT a first-position keyword. (3) quote in compile-ast: use raw-form→janet converter (map Jolt reader structs → Janet symbols) — don't re-analyze. (4) eval runs in default env, doesn't see `use`-imported symbols. FIX: embed fn values directly via core-fn-values table.
§
When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants — `(set ns-name ...)` on a `def` fails with "cannot set constant". This hit us in loader.janet ns-name extraction loop.
§
core-renames MUST match actual function names in core.janet. `"-"``core-sub` NOT `core--`. `"fn?"` was missing entirely (caused silent nil). Missing entries → symbol treated as unknown global, returns nil. When adding: grep core.janet for actual `(defn core-XXX)` name, add to BOTH core-renames (string table) and core-fn-values (fn value table).
§
Bare tuples in Janet's `eval` are function calls: `(eval [1 2 3])` tries to call `1` as function. Always emit `['tuple 1 2 3]` or `(tuple 1 2 3)` in data-structure emitter. Similarly, `(eval (try body (sym handler)))` fails because `catch` is not a Janet special form — must be `(try body ([sym] handler))`. Discovered during Phase 5/6 compiler work.
§
core-renames MUST match actual function names in core.janet. `"-"``core-sub` NOT `core--`. `"fn?"` was missing entirely (caused silent nil). Missing entries → symbol treated as unknown global, returns nil. When adding: grep core.janet for actual `(defn core-XXX)` name, add to BOTH core-renames (string table) and core-fn-values (fn value table).
§
Bare tuples in Janet's `eval` are function calls: `(eval [1 2 3])` tries to call `1` as function. Always emit `['tuple 1 2 3]` or `(tuple 1 2 3)` in data-structure emitter. Similarly, `(eval (try body (sym handler)))` fails because `catch` is not a Janet special form — must be `(try body ([sym] handler))`. Discovered during Phase 5/6 compiler work.

View file

@ -2,45 +2,47 @@
"jolt-bootstrap": {
"created_by": "agent",
"use_count": 9,
"view_count": 20,
"view_count": 22,
"patch_count": 10,
"last_used_at": "2026-06-02T18:45:23.336653+00:00",
"last_viewed_at": "2026-06-02T18:45:23.328485+00:00",
"last_viewed_at": "2026-06-02T20:34:59.785509+00:00",
"last_patched_at": "2026-06-02T03:44:45.935676+00:00",
"created_at": "2026-06-01T21:49:51.101718+00:00",
"state": "active",
"pinned": false
},
"jpm-build": {
"created_by": "agent",
"use_count": 0,
"view_count": 11,
"patch_count": 0,
"last_viewed_at": "2026-06-02T17:43:38.327259+00:00",
"created_at": "2026-06-01T20:56:39.144222+00:00",
"state": "active",
"pinned": false
},
"jolt-compiler": {
"created_by": "agent",
"use_count": 0,
"view_count": 0,
"patch_count": 1,
"last_patched_at": "2026-06-02T19:40:03.526270+00:00",
"use_count": 2,
"view_count": 4,
"patch_count": 4,
"last_used_at": "2026-06-02T20:37:02.987991+00:00",
"last_viewed_at": "2026-06-02T20:37:02.978817+00:00",
"last_patched_at": "2026-06-02T20:37:29.530250+00:00",
"created_at": "2026-06-02T17:54:38.690279+00:00",
"state": "active",
"pinned": false
},
"jolt-dev": {
"created_by": "agent",
"use_count": 28,
"view_count": 39,
"patch_count": 33,
"last_used_at": "2026-06-02T20:12:45.918073+00:00",
"last_viewed_at": "2026-06-02T20:12:45.904425+00:00",
"last_patched_at": "2026-06-02T20:13:13.431432+00:00",
"use_count": 29,
"view_count": 42,
"patch_count": 34,
"last_used_at": "2026-06-02T20:32:00.407408+00:00",
"last_viewed_at": "2026-06-02T20:34:59.800746+00:00",
"last_patched_at": "2026-06-02T20:33:05.356194+00:00",
"created_at": "2026-06-01T21:26:06.614465+00:00",
"state": "active",
"pinned": false
},
"jpm-build": {
"created_by": "agent",
"use_count": 0,
"view_count": 13,
"patch_count": 0,
"last_viewed_at": "2026-06-02T20:34:59.807762+00:00",
"created_at": "2026-06-01T20:56:39.144222+00:00",
"state": "active",
"pinned": false
}
}

View file

@ -70,14 +70,55 @@ Functions that need special mapping (name differs):
- `"-"``"core-sub"` (not `core--`)
- `"some"``core-some?` (shared with `core-some?`)
- `"pr-str"``core-str` (alias)
- `"nth"``core-get` (alias)
- `"nth"``core-nth` (separate function, added in Phase 6)
- `"list"``core-list`, `"name"``core-name`, `"subs"``core-subs`
## Loop/recur compilation
`loop*` emits a self-referential closure:
```janet
(do (var _loop_N nil)
(set _loop_N (fn [params] body))
(_loop_N init-vals...))
```
`recur` saves `:loop-name` in the AST (looked up from bindings `:jolt/current-loop`), then `emit-recur-expr` rewrites to `(loop-name arg1 arg2...)`.
In the string emitter, recur similarly emits `(loop-name arg ...)`.
## Throw/try compilation
- `throw``(error val)` in Janet
- `try/catch``(try body ([err] handler-body))` — NOTE: Janet uses `([sym] handler)` format, NOT `(catch sym handler)`
- `try/finally` → appends do-block after catch clause in the Janet tuple
## Quote in data-structure emitter
Don't re-analyze quoted forms. Use `raw-form->janet` to pass Jolt reader forms through verbatim to Janet's `quote`:
```
(emit-quote-expr expr) → ['quote (raw-form->janet expr)]
```
raw-form->janet converts symbols to Janet symbols, arrays/tuples recursively.
## Remaining ops (interpreter only)
`syntax-quote`, `set!`, `deftype`, `defmulti`, `defmethod` — these are stateful or complex and always use the interpreter path even in compile mode.
## Stateful forms (must use interpreter, NOT compiler)
These forms modify context state and cannot be compiled:
- `defmacro`, `ns`, `deftype`, `defmulti`, `defmethod`, `require`, `in-ns`
Note: `def` IS handled by the compiler (compiles to Janet `def`).
Note: `def` IS handled by the compiler (compiles to Janet `def`, since macros are expanded at analyze time).
## eval-string dispatch (compile mode)
```janet
(if (or (= head-name "defmacro") (= head-name "ns")
(= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod")
(= head-name "require") (= head-name "in-ns"))
(eval-form ctx @{} form) ; interpret
(compile-and-eval form ctx)) ; compile
## Adding a new op
@ -88,7 +129,12 @@ Note: `def` IS handled by the compiler (compiles to Janet `def`).
5. Add `core-fn-values` entry (Janet string name → actual fn value)
6. Add tests in `test/compiler-test.janet`
## Test patterns
## Test files
- `test/compiler-test.janet` — Phase 2-5 tests (source output + compile-eval + macro tests)
- `test/phase6-final.janet` — Phase 6 comprehensive compile-mode tests (47 assertions)
Run: `janet test/compiler-test.janet` or `janet test/phase6-final.janet` or `jpm test`
- Source output tests: `(assert (= "(expected)" (compile-str "(input)")) "label")`
- Round-trip tests: `(assert (= val (compile-eval-str "(input)")) "label")`

View file

@ -44,6 +44,36 @@ When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`.
Key design decision: **compile-and-eval emits Janet DATA STRUCTURES, not source strings**, because Janet's `eval` doesn't see `use`-imported symbols. `core-fn-values` table resolves Janet names to actual function values at compile time.
## Compiler Development
### Adding a new op to the compiler
1. Add match arm in `analyze-form` — maps Clojure form → AST node
2. Add `emit-*-str` for source-to-source path, then arm in `emit-ast` dispatch
3. Add `emit-*-expr` for data-structure path, then arm in `emit-expr` dispatch
4. Add tests
### Emitter patterns
**String emitter**: `(buffer/push buf "...")` → source text
**Data-structure emitter**: `['keyword val1 val2]` → eval-able tuples
### Janet eval gotchas
- Bare tuples are function calls — always use `['tuple ...]` or `(tuple ...)`
- `eval` scope: symbols from `(use ...)` not available — embed function VALUES
- Janet `try`: `(try body ([err] handler))` — not `(catch sym handler)`
- Core `-` maps to `core-sub` (NOT `core--`)
### Loop compilation
`(loop* [x 0] body-with-recur)``(do (var name nil) (set name (fn [x] body)) (name 0))`
Recur rewrites to `(loop-name arg...)` via `:jolt/current-loop` binding.
### Quote: use `raw-form->janet` converter, never re-analyze
### Macro expansion: pass ctx to `analyze-form`, check `resolve-macro`, expand + re-analyze
## Special Form Checklist
To add a new special form to the evaluator AND compiler:

View file

@ -56,7 +56,9 @@
nil)
stateful? (or (= head-name "defmacro") (= head-name "ns")
(= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod")
(= head-name "require") (= head-name "in-ns"))]
(= head-name "require") (= head-name "in-ns")
(= head-name "syntax-quote") (= head-name "set!")
(= head-name "var") (= head-name ".") (= head-name "new"))]
(if stateful?
(eval-form ctx @{} form)
(compile-and-eval form ctx)))

View file

@ -234,6 +234,90 @@
(++ loop-counter)
name))
# ============================================================
# Syntax-quote expansion
# ============================================================
(defn- sq-list?
"Check if form is a (unquote ...) or (unquote-splicing ...) call."
[form]
(and (array? form) (> (length form) 0)
(struct? (first form)) (= :symbol ((first form) :jolt/type))
(or (= "unquote" ((first form) :name))
(= "unquote-splicing" ((first form) :name)))))
(defn- sq-has-unquote?
"Check if any item in a collection is an unquote/unquote-splicing form."
[items]
(var found false) (var i 0)
(while (and (< i (length items)) (not found))
(if (sq-list? (in items i)) (set found true)) (++ i))
found)
(defn- sq-resolve-sym
"Qualify an unqualified symbol with the current namespace."
[sym-s ctx]
(if (and ctx (nil? (sym-s :ns)))
{:jolt/type :symbol :ns (ctx-current-ns ctx) :name (sym-s :name)}
sym-s))
(defn- syntax-quote-expand
"Expand a syntax-quoted form into a plain Clojure form.
Simple forms are wrapped in (quote ...). Forms with unquote produce
(concat (list ...) ...) calls."
[form ctx]
(cond
# unquote → just the inner expression
(and (array? form) (> (length form) 0)
(struct? (first form)) (= :symbol ((first form) :jolt/type))
(= "unquote" ((first form) :name)))
(in form 1)
# unquote-splicing → just the inner expression
(and (array? form) (> (length form) 0)
(struct? (first form)) (= :symbol ((first form) :jolt/type))
(= "unquote-splicing" ((first form) :name)))
(in form 1)
# Literals → (quote literal)
(or (nil? form) (= true form) (= false form)
(number? form) (string? form) (keyword? form))
[{:jolt/type :symbol :ns nil :name "quote"} form]
# Symbols → (quote resolved-symbol)
(and (struct? form) (= :symbol (form :jolt/type)))
[{:jolt/type :symbol :ns nil :name "quote"} (sq-resolve-sym form ctx)]
# Lists/arrays with unquote → (concat (list ...) (list ...) ...)
(array? form)
(if (sq-has-unquote? form)
(let [items (map |(syntax-quote-expand $ ctx) form)
concat-args @[]]
(each item items
(array/push concat-args
[{:jolt/type :symbol :ns nil :name "list"} item]))
(if (> (length concat-args) 1)
(tuple ;(array/insert concat-args 0
{:jolt/type :symbol :ns nil :name "concat"}))
(in concat-args 0)))
[{:jolt/type :symbol :ns nil :name "quote"} form])
# Vectors → (vec (concat (list ...) ...))
(tuple? form)
(if (sq-has-unquote? form)
(let [items (map |(syntax-quote-expand $ ctx) form)
concat-args @[]]
(each item items
(array/push concat-args
[{:jolt/type :symbol :ns nil :name "list"} item]))
[{:jolt/type :symbol :ns nil :name "vec"}
(tuple ;(array/insert concat-args 0
{:jolt/type :symbol :ns nil :name "concat"}))])
[{:jolt/type :symbol :ns nil :name "quote"} form])
# Default → (quote form)
[{:jolt/type :symbol :ns nil :name "quote"} form]))
# ============================================================
# Analyzer
# ============================================================
@ -553,8 +637,32 @@
(defn- emit-map-str [form buf] (buffer/push buf (string form)))
(defn- raw-form->janet
"Convert a Jolt reader form to a Janet data structure for quoting."
[form]
(cond
(and (struct? form) (= :symbol (form :jolt/type)))
(if (form :ns)
(symbol (string (form :ns) "/" (form :name)))
(symbol (form :name)))
(array? form)
(tuple/slice (tuple ;(map raw-form->janet form)))
(tuple? form)
(tuple/slice (tuple ;(map raw-form->janet form)))
form))
(defn- emit-quote-str [expr buf]
(buffer/push buf "'") (emit-ast (analyze-form expr @{}) buf))
(buffer/push buf "'")
(def janet-val (raw-form->janet expr))
(cond
(symbol? janet-val) (buffer/push buf (string janet-val))
(number? janet-val) (buffer/push buf (string janet-val))
(string? janet-val) (do (buffer/push buf "\"") (buffer/push buf janet-val) (buffer/push buf "\""))
(keyword? janet-val) (do (buffer/push buf ":") (buffer/push buf (string janet-val)))
(nil? janet-val) (buffer/push buf "nil")
(= true janet-val) (buffer/push buf "true")
(= false janet-val) (buffer/push buf "false")
(buffer/push buf (string janet-val))))
(set emit-ast
(fn [ast buf]
@ -677,18 +785,6 @@
(defn- emit-map-expr [form] form)
(defn- raw-form->janet
"Convert a Jolt reader form to a Janet data structure for quoting."
[form]
(cond
(and (struct? form) (= :symbol (form :jolt/type)))
(symbol (form :name))
(array? form)
(tuple/slice (tuple ;(map raw-form->janet form)))
(tuple? form)
(tuple/slice (tuple ;(map raw-form->janet form)))
form))
(defn- emit-quote-expr [expr]
['quote (raw-form->janet expr)])