diff --git a/.dirge/memory/MEMORY.md b/.dirge/memory/MEMORY.md index e059137..9dd4ff2 100644 --- a/.dirge/memory/MEMORY.md +++ b/.dirge/memory/MEMORY.md @@ -1,7 +1,37 @@ Janet's `eval` runs in Janet's default environment and does NOT have access to symbols imported via `(use ...)` in the calling file. `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. FIX: emit Janet data structures where function VALUES are embedded directly (e.g. `[core-inc 1]`) rather than source strings `"(core-inc 1)"`. The `core-fn-values` table resolves Janet symbol names to actual function values at compile time. § -Jolt Compiler Architecture (Phases 1-6, dfa9874→c366963): Two-phase — analyze-form (Clojure form → annotated AST) → emit-ast (→ Janet source string) or emit-expr (→ Janet data structures for eval). analyze-form takes [form bindings &opt ctx]; ctx needed for macro expansion. Symbol classification: bindings first (:local), then core-renames (:core-symbol), then plain (:symbol). Two emitter paths: string (compile-form) and data structures (compile-ast). Core fn values resolved via core-fn-values table. compile-and-eval takes [form ctx]; pass nil for no macro ctx. Naming: Clojure - → core-sub. Key bugs found: emit-vector-expr must wrap with (tuple ...) — bare tuples eval as fn calls. make-symbol must treat / at pos 0 as unqualified. raw-form->janet converter for quote (don't re-analyze). core-renames entry for fn? was missing. +Jolt Compiler Architecture (Phases 1-6, dfa9874→1de109f): Two-phase — analyze-form (Clojure form → annotated AST) → emit-ast (→ Janet source string) or emit-expr (→ Janet data structures for eval). analyze-form takes [form bindings &opt ctx]; ctx needed for macro expansion. Symbol classification: bindings first (:local), then core-renames (:core-symbol), then plain (:symbol). Two emitter paths: string (compile-form) and data structures (compile-ast). Core fn values resolved via core-fn-values table. compile-and-eval takes [form ctx]; pass nil for no macro ctx. + +Key naming/facts: +- Clojure - → core-sub (NOT core--) +- core-nth did not exist — had to add both the function and core-bindings entry +- Missing from core-renames early: fn?, list, name, subs +- Bare tuples in Janet eval → treated as function calls. Always emit (tuple ...) or ['tuple ...] +- make-symbol: / at position 0 means unqualified symbol (was parsing empty ns) +- raw-form->janet converter for quote: don't re-analyze quoted forms, pass through verbatim +- emit-try-expr: Janet format is (try body ([err] handler)) not (try body (catch sym handler)) +- Loop compilation: (do (var _loop_N nil) (set _loop_N (fn [params] body)) (_loop_N init-vals...)) +- Recur compilation: rewrites to (loop-name arg1 arg2...) via :loop-name in AST + +eval-string dispatch: When :compile? true, stateful forms (defmacro, ns, deftype, defmulti, defmethod, require, in-ns) use interpreter. All others (def, macros like defn) go through compile-and-eval. Macros expanded at analyze time via resolve-macro. + +Remaining: syntax-quote, set! compiler support. deftype/defmulti/defmethod routed to interpreter. § -eval-string dispatch (api.janet): When :compile? is true, stateful forms (defmacro, ns, deftype, defmulti, defmethod, require, in-ns) always use interpreter. All others (def, defn via macros, pure expressions) go through compile-and-eval. defn bug was caused by macros not being expanded — fixed in Phase 4 by routing macros through compiler's analyze-form which expands them at analyze time. +Jolt Compiler Architecture (Phases 1-6, dfa9874→1de109f): Two-phase — analyze-form (Clojure form → annotated AST) → emit-ast (→ Janet source string) or emit-expr (→ Janet data structures for eval). analyze-form takes [form bindings &opt ctx]; ctx needed for macro expansion. Symbol classification: bindings first (:local), then core-renames (:core-symbol), then plain (:symbol). Two emitter paths: string (compile-form) and data structures (compile-ast). Core fn values resolved via core-fn-values table. compile-and-eval takes [form ctx]; pass nil for no macro ctx. + +Key naming/facts: +- Clojure - → core-sub (NOT core--) +- core-nth did not exist — had to add both the function and core-bindings entry +- Missing from core-renames early: fn?, list, name, subs +- Bare tuples in Janet eval → treated as function calls. Always emit (tuple ...) or ['tuple ...] +- make-symbol: / at position 0 means unqualified symbol (was parsing empty ns) +- raw-form->janet converter for quote: don't re-analyze quoted forms, pass through verbatim +- emit-try-expr: Janet format is (try body ([err] handler)) not (try body (catch sym handler)) +- Loop compilation: (do (var _loop_N nil) (set _loop_N (fn [params] body)) (_loop_N init-vals...)) +- Recur compilation: rewrites to (loop-name arg1 arg2...) via :loop-name in AST + +eval-string dispatch: When :compile? true, stateful forms (defmacro, ns, deftype, defmulti, defmethod, require, in-ns) use interpreter. All others (def, macros like defn) go through compile-and-eval. Macros expanded at analyze time via resolve-macro. + +Remaining: syntax-quote, set! compiler support. deftype/defmulti/defmethod routed to interpreter. Git push needs manual approval. § -Phase 6: 47 comprehensive compile-mode tests in test/phase6-final.janet. Collections, math, predicates, comparison, seq ops (map/filter/reduce/take/drop), special forms (let/if/loop/try/quote), macros (defn/when/and/or/fn/if-let), complex nesting. All 317 tests pass. Remaining: syntax-quote, set! compiler support. deftype, defmulti/defmethod routed to interpreter (stateful). +Phase 6: 47 comprehensive compile-mode tests in test/phase6-final.janet. Collections, math, predicates, comparison, seq ops (map/filter/reduce/take/drop), special forms (let/if/loop/try/quote), macros (defn/when/and/or/fn/if-let), complex nesting. 58 assertions. All 317 tests pass, 0 failures. Remaining: syntax-quote, set! compiler support. deftype, defmulti/defmethod routed to interpreter (stateful). diff --git a/.dirge/memory/PITFALLS.md b/.dirge/memory/PITFALLS.md index 95f166b..dbe7929 100644 --- a/.dirge/memory/PITFALLS.md +++ b/.dirge/memory/PITFALLS.md @@ -1,5 +1,9 @@ -Janet eval data-structure constraints: (1) Bare tuples eval as fn calls — use `['tuple val1 val2]` not `[val1 val2]` for vector values. (2) Janet try: `[(tuple ;[err-sym]) handler-body]` NOT `(catch [err] body)` — catch is NOT a first-position keyword. (3) quote in compile-ast: use raw-form→janet converter (map Jolt reader structs → Janet symbols) — don't re-analyze. (4) eval runs in default env, doesn't see `use`-imported symbols. FIX: embed fn values directly via core-fn-values table. -§ When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants — `(set ns-name ...)` on a `def` fails with "cannot set constant". This hit us in loader.janet ns-name extraction loop. § core-renames MUST match actual function names in core.janet. `"-"`→`core-sub` NOT `core--`. `"fn?"` was missing entirely (caused silent nil). Missing entries → symbol treated as unknown global, returns nil. When adding: grep core.janet for actual `(defn core-XXX)` name, add to BOTH core-renames (string table) and core-fn-values (fn value table). +§ +Bare tuples in Janet's `eval` are function calls: `(eval [1 2 3])` tries to call `1` as function. Always emit `['tuple 1 2 3]` or `(tuple 1 2 3)` in data-structure emitter. Similarly, `(eval (try body (sym handler)))` fails because `catch` is not a Janet special form — must be `(try body ([sym] handler))`. Discovered during Phase 5/6 compiler work. +§ +core-renames MUST match actual function names in core.janet. `"-"`→`core-sub` NOT `core--`. `"fn?"` was missing entirely (caused silent nil). Missing entries → symbol treated as unknown global, returns nil. When adding: grep core.janet for actual `(defn core-XXX)` name, add to BOTH core-renames (string table) and core-fn-values (fn value table). +§ +Bare tuples in Janet's `eval` are function calls: `(eval [1 2 3])` tries to call `1` as function. Always emit `['tuple 1 2 3]` or `(tuple 1 2 3)` in data-structure emitter. Similarly, `(eval (try body (sym handler)))` fails because `catch` is not a Janet special form — must be `(try body ([sym] handler))`. Discovered during Phase 5/6 compiler work. diff --git a/.dirge/skills/.usage.json b/.dirge/skills/.usage.json index 5a52239..267a885 100644 --- a/.dirge/skills/.usage.json +++ b/.dirge/skills/.usage.json @@ -2,45 +2,47 @@ "jolt-bootstrap": { "created_by": "agent", "use_count": 9, - "view_count": 20, + "view_count": 22, "patch_count": 10, "last_used_at": "2026-06-02T18:45:23.336653+00:00", - "last_viewed_at": "2026-06-02T18:45:23.328485+00:00", + "last_viewed_at": "2026-06-02T20:34:59.785509+00:00", "last_patched_at": "2026-06-02T03:44:45.935676+00:00", "created_at": "2026-06-01T21:49:51.101718+00:00", "state": "active", "pinned": false }, - "jpm-build": { - "created_by": "agent", - "use_count": 0, - "view_count": 11, - "patch_count": 0, - "last_viewed_at": "2026-06-02T17:43:38.327259+00:00", - "created_at": "2026-06-01T20:56:39.144222+00:00", - "state": "active", - "pinned": false - }, "jolt-compiler": { "created_by": "agent", - "use_count": 0, - "view_count": 0, - "patch_count": 1, - "last_patched_at": "2026-06-02T19:40:03.526270+00:00", + "use_count": 2, + "view_count": 4, + "patch_count": 4, + "last_used_at": "2026-06-02T20:37:02.987991+00:00", + "last_viewed_at": "2026-06-02T20:37:02.978817+00:00", + "last_patched_at": "2026-06-02T20:37:29.530250+00:00", "created_at": "2026-06-02T17:54:38.690279+00:00", "state": "active", "pinned": false }, "jolt-dev": { "created_by": "agent", - "use_count": 28, - "view_count": 39, - "patch_count": 33, - "last_used_at": "2026-06-02T20:12:45.918073+00:00", - "last_viewed_at": "2026-06-02T20:12:45.904425+00:00", - "last_patched_at": "2026-06-02T20:13:13.431432+00:00", + "use_count": 29, + "view_count": 42, + "patch_count": 34, + "last_used_at": "2026-06-02T20:32:00.407408+00:00", + "last_viewed_at": "2026-06-02T20:34:59.800746+00:00", + "last_patched_at": "2026-06-02T20:33:05.356194+00:00", "created_at": "2026-06-01T21:26:06.614465+00:00", "state": "active", "pinned": false + }, + "jpm-build": { + "created_by": "agent", + "use_count": 0, + "view_count": 13, + "patch_count": 0, + "last_viewed_at": "2026-06-02T20:34:59.807762+00:00", + "created_at": "2026-06-01T20:56:39.144222+00:00", + "state": "active", + "pinned": false } } \ No newline at end of file diff --git a/.dirge/skills/jolt-compiler/SKILL.md b/.dirge/skills/jolt-compiler/SKILL.md index 9f3be90..7223b88 100644 --- a/.dirge/skills/jolt-compiler/SKILL.md +++ b/.dirge/skills/jolt-compiler/SKILL.md @@ -70,14 +70,55 @@ Functions that need special mapping (name differs): - `"-"` → `"core-sub"` (not `core--`) - `"some"` → `core-some?` (shared with `core-some?`) - `"pr-str"` → `core-str` (alias) -- `"nth"` → `core-get` (alias) +- `"nth"` → `core-nth` (separate function, added in Phase 6) +- `"list"` → `core-list`, `"name"` → `core-name`, `"subs"` → `core-subs` + +## Loop/recur compilation + +`loop*` emits a self-referential closure: +```janet +(do (var _loop_N nil) + (set _loop_N (fn [params] body)) + (_loop_N init-vals...)) +``` + +`recur` saves `:loop-name` in the AST (looked up from bindings `:jolt/current-loop`), then `emit-recur-expr` rewrites to `(loop-name arg1 arg2...)`. + +In the string emitter, recur similarly emits `(loop-name arg ...)`. + +## Throw/try compilation + +- `throw` → `(error val)` in Janet +- `try/catch` → `(try body ([err] handler-body))` — NOTE: Janet uses `([sym] handler)` format, NOT `(catch sym handler)` +- `try/finally` → appends do-block after catch clause in the Janet tuple + +## Quote in data-structure emitter + +Don't re-analyze quoted forms. Use `raw-form->janet` to pass Jolt reader forms through verbatim to Janet's `quote`: +``` +(emit-quote-expr expr) → ['quote (raw-form->janet expr)] +``` +raw-form->janet converts symbols to Janet symbols, arrays/tuples recursively. + +## Remaining ops (interpreter only) + +`syntax-quote`, `set!`, `deftype`, `defmulti`, `defmethod` — these are stateful or complex and always use the interpreter path even in compile mode. ## Stateful forms (must use interpreter, NOT compiler) These forms modify context state and cannot be compiled: - `defmacro`, `ns`, `deftype`, `defmulti`, `defmethod`, `require`, `in-ns` -Note: `def` IS handled by the compiler (compiles to Janet `def`). +Note: `def` IS handled by the compiler (compiles to Janet `def`, since macros are expanded at analyze time). + +## eval-string dispatch (compile mode) + +```janet +(if (or (= head-name "defmacro") (= head-name "ns") + (= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod") + (= head-name "require") (= head-name "in-ns")) + (eval-form ctx @{} form) ; interpret + (compile-and-eval form ctx)) ; compile ## Adding a new op @@ -88,7 +129,12 @@ Note: `def` IS handled by the compiler (compiles to Janet `def`). 5. Add `core-fn-values` entry (Janet string name → actual fn value) 6. Add tests in `test/compiler-test.janet` -## Test patterns +## Test files + +- `test/compiler-test.janet` — Phase 2-5 tests (source output + compile-eval + macro tests) +- `test/phase6-final.janet` — Phase 6 comprehensive compile-mode tests (47 assertions) + +Run: `janet test/compiler-test.janet` or `janet test/phase6-final.janet` or `jpm test` - Source output tests: `(assert (= "(expected)" (compile-str "(input)")) "label")` - Round-trip tests: `(assert (= val (compile-eval-str "(input)")) "label")` diff --git a/.dirge/skills/jolt-dev/SKILL.md b/.dirge/skills/jolt-dev/SKILL.md index e040ce7..0e6518a 100644 --- a/.dirge/skills/jolt-dev/SKILL.md +++ b/.dirge/skills/jolt-dev/SKILL.md @@ -44,6 +44,36 @@ When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. Key design decision: **compile-and-eval emits Janet DATA STRUCTURES, not source strings**, because Janet's `eval` doesn't see `use`-imported symbols. `core-fn-values` table resolves Janet names to actual function values at compile time. +## Compiler Development + +### Adding a new op to the compiler + +1. Add match arm in `analyze-form` — maps Clojure form → AST node +2. Add `emit-*-str` for source-to-source path, then arm in `emit-ast` dispatch +3. Add `emit-*-expr` for data-structure path, then arm in `emit-expr` dispatch +4. Add tests + +### Emitter patterns + +**String emitter**: `(buffer/push buf "...")` → source text +**Data-structure emitter**: `['keyword val1 val2]` → eval-able tuples + +### Janet eval gotchas + +- Bare tuples are function calls — always use `['tuple ...]` or `(tuple ...)` +- `eval` scope: symbols from `(use ...)` not available — embed function VALUES +- Janet `try`: `(try body ([err] handler))` — not `(catch sym handler)` +- Core `-` maps to `core-sub` (NOT `core--`) + +### Loop compilation + +`(loop* [x 0] body-with-recur)` → `(do (var name nil) (set name (fn [x] body)) (name 0))` +Recur rewrites to `(loop-name arg...)` via `:jolt/current-loop` binding. + +### Quote: use `raw-form->janet` converter, never re-analyze + +### Macro expansion: pass ctx to `analyze-form`, check `resolve-macro`, expand + re-analyze + ## Special Form Checklist To add a new special form to the evaluator AND compiler: diff --git a/src/jolt/api.janet b/src/jolt/api.janet index 4483b3f..c50f035 100644 --- a/src/jolt/api.janet +++ b/src/jolt/api.janet @@ -56,7 +56,9 @@ nil) stateful? (or (= head-name "defmacro") (= head-name "ns") (= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod") - (= head-name "require") (= head-name "in-ns"))] + (= head-name "require") (= head-name "in-ns") + (= head-name "syntax-quote") (= head-name "set!") + (= head-name "var") (= head-name ".") (= head-name "new"))] (if stateful? (eval-form ctx @{} form) (compile-and-eval form ctx))) diff --git a/src/jolt/compiler.janet b/src/jolt/compiler.janet index a931b98..b806c94 100644 --- a/src/jolt/compiler.janet +++ b/src/jolt/compiler.janet @@ -234,6 +234,90 @@ (++ loop-counter) name)) +# ============================================================ +# Syntax-quote expansion +# ============================================================ + +(defn- sq-list? + "Check if form is a (unquote ...) or (unquote-splicing ...) call." + [form] + (and (array? form) (> (length form) 0) + (struct? (first form)) (= :symbol ((first form) :jolt/type)) + (or (= "unquote" ((first form) :name)) + (= "unquote-splicing" ((first form) :name))))) + +(defn- sq-has-unquote? + "Check if any item in a collection is an unquote/unquote-splicing form." + [items] + (var found false) (var i 0) + (while (and (< i (length items)) (not found)) + (if (sq-list? (in items i)) (set found true)) (++ i)) + found) + +(defn- sq-resolve-sym + "Qualify an unqualified symbol with the current namespace." + [sym-s ctx] + (if (and ctx (nil? (sym-s :ns))) + {:jolt/type :symbol :ns (ctx-current-ns ctx) :name (sym-s :name)} + sym-s)) + +(defn- syntax-quote-expand + "Expand a syntax-quoted form into a plain Clojure form. + Simple forms are wrapped in (quote ...). Forms with unquote produce + (concat (list ...) ...) calls." + [form ctx] + (cond + # unquote → just the inner expression + (and (array? form) (> (length form) 0) + (struct? (first form)) (= :symbol ((first form) :jolt/type)) + (= "unquote" ((first form) :name))) + (in form 1) + + # unquote-splicing → just the inner expression + (and (array? form) (> (length form) 0) + (struct? (first form)) (= :symbol ((first form) :jolt/type)) + (= "unquote-splicing" ((first form) :name))) + (in form 1) + + # Literals → (quote literal) + (or (nil? form) (= true form) (= false form) + (number? form) (string? form) (keyword? form)) + [{:jolt/type :symbol :ns nil :name "quote"} form] + + # Symbols → (quote resolved-symbol) + (and (struct? form) (= :symbol (form :jolt/type))) + [{:jolt/type :symbol :ns nil :name "quote"} (sq-resolve-sym form ctx)] + + # Lists/arrays with unquote → (concat (list ...) (list ...) ...) + (array? form) + (if (sq-has-unquote? form) + (let [items (map |(syntax-quote-expand $ ctx) form) + concat-args @[]] + (each item items + (array/push concat-args + [{:jolt/type :symbol :ns nil :name "list"} item])) + (if (> (length concat-args) 1) + (tuple ;(array/insert concat-args 0 + {:jolt/type :symbol :ns nil :name "concat"})) + (in concat-args 0))) + [{:jolt/type :symbol :ns nil :name "quote"} form]) + + # Vectors → (vec (concat (list ...) ...)) + (tuple? form) + (if (sq-has-unquote? form) + (let [items (map |(syntax-quote-expand $ ctx) form) + concat-args @[]] + (each item items + (array/push concat-args + [{:jolt/type :symbol :ns nil :name "list"} item])) + [{:jolt/type :symbol :ns nil :name "vec"} + (tuple ;(array/insert concat-args 0 + {:jolt/type :symbol :ns nil :name "concat"}))]) + [{:jolt/type :symbol :ns nil :name "quote"} form]) + + # Default → (quote form) + [{:jolt/type :symbol :ns nil :name "quote"} form])) + # ============================================================ # Analyzer # ============================================================ @@ -553,8 +637,32 @@ (defn- emit-map-str [form buf] (buffer/push buf (string form))) +(defn- raw-form->janet + "Convert a Jolt reader form to a Janet data structure for quoting." + [form] + (cond + (and (struct? form) (= :symbol (form :jolt/type))) + (if (form :ns) + (symbol (string (form :ns) "/" (form :name))) + (symbol (form :name))) + (array? form) + (tuple/slice (tuple ;(map raw-form->janet form))) + (tuple? form) + (tuple/slice (tuple ;(map raw-form->janet form))) + form)) + (defn- emit-quote-str [expr buf] - (buffer/push buf "'") (emit-ast (analyze-form expr @{}) buf)) + (buffer/push buf "'") + (def janet-val (raw-form->janet expr)) + (cond + (symbol? janet-val) (buffer/push buf (string janet-val)) + (number? janet-val) (buffer/push buf (string janet-val)) + (string? janet-val) (do (buffer/push buf "\"") (buffer/push buf janet-val) (buffer/push buf "\"")) + (keyword? janet-val) (do (buffer/push buf ":") (buffer/push buf (string janet-val))) + (nil? janet-val) (buffer/push buf "nil") + (= true janet-val) (buffer/push buf "true") + (= false janet-val) (buffer/push buf "false") + (buffer/push buf (string janet-val)))) (set emit-ast (fn [ast buf] @@ -677,18 +785,6 @@ (defn- emit-map-expr [form] form) -(defn- raw-form->janet - "Convert a Jolt reader form to a Janet data structure for quoting." - [form] - (cond - (and (struct? form) (= :symbol (form :jolt/type))) - (symbol (form :name)) - (array? form) - (tuple/slice (tuple ;(map raw-form->janet form))) - (tuple? form) - (tuple/slice (tuple ;(map raw-form->janet form))) - form)) - (defn- emit-quote-expr [expr] ['quote (raw-form->janet expr)])