From a8c453183f2370fe221f903749168c459416e05f Mon Sep 17 00:00:00 2001 From: Yogthos Date: Tue, 2 Jun 2026 15:43:08 -0400 Subject: [PATCH] Phase 4: Macro expansion in compiler - resolve-macro: resolve symbols to macro vars via ctx - Macro expansion in analyze-form: detects macro heads, expands, re-analyzes - compile-ast: emits Janet data structures with resolved core fn values - compile-and-eval uses compile-ast (no source parse roundtrip) - eval-string routes macros through compiler (expanded at analyze time) - Fix - mapping: core-sub (core-- doesn't exist) - All 317 tests pass + 6 new Phase 4 macro tests --- .dirge/memory/MEMORY.md | 6 +- .dirge/memory/PITFALLS.md | 6 +- .dirge/skills/.usage.json | 35 +-- .dirge/skills/jolt-compiler/SKILL.md | 137 +++++----- .dirge/skills/jolt-dev/SKILL.md | 9 +- src/jolt/api.janet | 18 +- src/jolt/compiler.janet | 389 ++++++++++++++++++++------- src/jolt/loader.janet | 5 +- test/compiler-test.janet | 31 ++- 9 files changed, 448 insertions(+), 188 deletions(-) diff --git a/.dirge/memory/MEMORY.md b/.dirge/memory/MEMORY.md index e790f18..19fa6e7 100644 --- a/.dirge/memory/MEMORY.md +++ b/.dirge/memory/MEMORY.md @@ -1,5 +1,5 @@ -Compiler Phases 1-3 COMPLETE. src/jolt/compiler.janet (342 lines): analyze-form→emit-ast two-phase. Phase 2: symbol classification (locals→core→symbol, shadowing works). Phase 3: compile-and-eval via parser→consume→eof→produce→eval pipeline, :compile? flag in types.janet context, loader.janet (80 lines) for namespace loading, api.janet updated (compile-string, compile-file, compile-aware eval-string). Stateful forms (def, defmacro, ns, deftype, defmulti, defmethod, require, in-ns) always route to interpreter. All 317 tests pass. Next: Phase 4 (macro integration), Phase 5 (remaining ops: loop/recur, try/throw, quote, syntax-quote, set!, .), Phase 6 (benchmarks). +Compiler in `src/jolt/compiler.janet` (~470 lines). Three emit modes: 1) compile-form → Janet source string (debug/display) 2) compile-ast → Janet data structures with resolved fn values (for eval) 3) compile-and-eval → compile-ast + eval. core-renames maps Clojure→Janet STRING names. core-fn-values maps Janet string names→actual function VALUES (used by compile-ast). analyze-form [form bindings ctx] — ctx optional for macro expansion. Macro expansion: when head symbol resolves to macro var, apply macro fn to args and re-analyze expanded form. Ops: const, do, if, def, fn*, let*, invoke, symbol (local/core/qualified), quote, vector, map. Symbol classification: locals → core-symbol → symbol (shadowing works). compile-form and compile-ast accept optional ctx arg. § -Compiler is in `src/jolt/compiler.janet` (342 lines). Two-phase: `analyze-form [form bindings]` → annotated AST with `:op` keys, `emit-ast` dispatches on `:op` → Janet source string via StringBuffer. `compile-form` for source output, `compile-and-eval` for direct eval via `eval-janet-source` (parser/new→consume→eof→produce→eval pipeline). `core-renames` table maps Clojure→Janet names. Ops covered: const, do, if, def, fn*, let*, invoke, symbol (local/core-symbol/qualified classification), quote. Symbol classification order: locals → core-symbol → symbol (shadowing works). Stateful forms (def, defmacro, etc.) must use interpreter path, not compile-and-eval. +Compiler tests in `test/compiler-test.janet` (172 lines, 11 test groups). Groups: 1-literals, 2-do, 3-if, 4-def, 5-fn, 6-let, 7-invoke, 8-local classification, 9-compile-and-eval round-trip, 10-compile flag, 11-macro expansion (defn, when, let, fn, and/or via compile?). All 317 tests pass. § -Compiler tests in `test/compiler-test.janet` (143 lines, 10 test groups). Phase 1: 7 groups, 19 string-output assertions. Phase 2: local classification (3 assertions: shadowing, param shadowing, nested let). Phase 3: compile-and-eval round-trip (9 assertions) + :compile? flag tests (7 assertions). All 317 tests pass. Run with `janet test/compiler-test.janet` or `jpm test`. +Janet's `eval` runs in Janet's default environment and does NOT have access to symbols imported via `(use ...)` in the calling file. `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. FIX: emit Janet data structures where function VALUES are embedded directly (e.g. `[core-inc 1]`) rather than source strings `"(core-inc 1)"`. The `core-fn-values` table resolves Janet symbol names to actual function values at compile time. diff --git a/.dirge/memory/PITFALLS.md b/.dirge/memory/PITFALLS.md index 625dd33..b0de493 100644 --- a/.dirge/memory/PITFALLS.md +++ b/.dirge/memory/PITFALLS.md @@ -1,5 +1,5 @@ -Janet LSP produces false positives on `.janet` files — it doesn't understand Janet syntax (thinks docstring lines are unresolved symbols, doesn't know `declare-project`/`declare-source` macros, etc.). These are pre-existing and should be ignored — they don't affect runtime correctness. -§ -Janet `(parse s)` returns `[symbol, error-position]` for forms — e.g., `(parse "(+ 1 2)")` gives `[+ 1]`, not a parsed tuple. For proper evaluation of special forms, must use the parser pipeline: `parser/new` → `parser/consume` → `parser/eof` → `parser/produce` → `eval`. Without `parser/eof`, `produce` returns nil for simple forms like numbers. Janet's `eval` also doesn't support special forms natively in data structures — `(eval [if true 1 2])` fails with "unknown symbol if". Must parse strings, not eval tuples. +Janet eval scope & source-to-source failure: `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. Janet's `eval` runs in the default environment and doesn't see `use`-imported symbols. NEVER emit Janet source strings for eval. Instead emit Janet data structures where function VALUES are embedded directly: `[core-inc 1]` not `"(core-inc 1)"`. Use a `core-fn-values` table to resolve names at compile time. The `(parse s)` function returns `[symbol, error-position]` — must use parser pipeline for proper eval: `parser/new→consume→eof→produce→eval`. Without `parser/eof`, numbers/literals return nil from produce. Janet's `eval` also doesn't support special forms natively in data structures — `(eval [if true 1 2])` fails with "unknown symbol if". Must parse strings, not eval tuples. § When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants — `(set ns-name ...)` on a `def` fails with "cannot set constant". This hit us in loader.janet ns-name extraction loop. +§ +core-renames MUST match actual function names defined in core.janet. `"-"` maps to `core-sub` NOT `core--`. `"not"` maps to `core-not` (defined as `(defn core-not ...)`). Missing entries cause silent nil returns. When adding to core-renames, grep core.janet for the actual `(defn core-XXX ...)` or `(def core-XXX ...)` name. Then add matching entry to core-fn-values. diff --git a/.dirge/skills/.usage.json b/.dirge/skills/.usage.json index 6b357ed..9e8da8f 100644 --- a/.dirge/skills/.usage.json +++ b/.dirge/skills/.usage.json @@ -1,22 +1,12 @@ { - "jpm-build": { - "created_by": "agent", - "use_count": 0, - "view_count": 11, - "patch_count": 0, - "last_viewed_at": "2026-06-02T17:43:38.327259+00:00", - "created_at": "2026-06-01T20:56:39.144222+00:00", - "state": "active", - "pinned": false - }, "jolt-dev": { "created_by": "agent", - "use_count": 25, - "view_count": 36, - "patch_count": 31, - "last_used_at": "2026-06-02T19:13:37.620054+00:00", - "last_viewed_at": "2026-06-02T19:13:37.612317+00:00", - "last_patched_at": "2026-06-02T19:14:06.338745+00:00", + "use_count": 26, + "view_count": 37, + "patch_count": 32, + "last_used_at": "2026-06-02T19:38:51.833410+00:00", + "last_viewed_at": "2026-06-02T19:38:51.825602+00:00", + "last_patched_at": "2026-06-02T19:39:44.919664+00:00", "created_at": "2026-06-01T21:26:06.614465+00:00", "state": "active", "pinned": false @@ -25,7 +15,8 @@ "created_by": "agent", "use_count": 0, "view_count": 0, - "patch_count": 0, + "patch_count": 1, + "last_patched_at": "2026-06-02T19:40:03.526270+00:00", "created_at": "2026-06-02T17:54:38.690279+00:00", "state": "active", "pinned": false @@ -41,5 +32,15 @@ "created_at": "2026-06-01T21:49:51.101718+00:00", "state": "active", "pinned": false + }, + "jpm-build": { + "created_by": "agent", + "use_count": 0, + "view_count": 11, + "patch_count": 0, + "last_viewed_at": "2026-06-02T17:43:38.327259+00:00", + "created_at": "2026-06-01T20:56:39.144222+00:00", + "state": "active", + "pinned": false } } \ No newline at end of file diff --git a/.dirge/skills/jolt-compiler/SKILL.md b/.dirge/skills/jolt-compiler/SKILL.md index 157931a..9f3be90 100644 --- a/.dirge/skills/jolt-compiler/SKILL.md +++ b/.dirge/skills/jolt-compiler/SKILL.md @@ -1,82 +1,97 @@ --- -description: Jolt compiler architecture and implementation plan +triggers: + - "compile jolt" + - "jolt compiler" + - "Clojure to Janet compilation" + - "add new op to compiler" + - "fix compiler" --- # Jolt Compiler -Two-phase source-to-source compiler: Clojure forms → annotated AST → Janet source → Janet bytecode. +Source-to-source Clojure→Janet compiler. Two-phase: analyze-form (classify + macro expand) → emit-ast (generate). ## Architecture ``` -Clojure source → Reader → raw AST - ↓ - analyze-form (classify symbols, produce :op AST) - ↓ - emit* dispatch (generate Janet source string) - ↓ - Janet compile → bytecode +Clojure form → analyze-form [form bindings ctx] → AST {:op ...} + ↓ (if head = macro var) + expand → re-analyze expanded form + ↓ + emit-ast (source string) or emit-expr (data structure) ``` -Follows CLJS `cljs.analyzer` / `cljs.compiler` pattern. +Three public entry points: +- `(compile-form form &opt ctx)` → Janet source string (debug/display) +- `(compile-ast form &opt ctx)` → Janet data structure (for eval) +- `(compile-and-eval form ctx)` → compile-ast + eval -## Key decisions +## Why data structures, not source strings -- **Target**: Janet source text (fed to Janet's `compile`), not direct bytecode. Simpler, debuggable, portable across Janet versions. -- **Mode gating**: Opt-in per context via `:compile?` flag on `init`. `eval-string` still interprets unless opted in. -- **Caching**: In-memory bytecode cache in context first. Disk persistence (`.jimage` files) as follow-up. - -## AST ops (CLJS subset + Jolt-specific) - -Core: `const`, `var`, `local`, `binding`, `if`, `do`, `let`, `loop`, `recur`, `fn`, `fn-method`, `def`, `invoke`, `quote`, `try`, `throw`, `set!`, `new`, `host-field`, `host-call` - -Jolt-specific: `deftype`, `defmulti`, `defmethod`, `syntax-quote` - -## File layout - -| File | Purpose | -|------|---------| -| `src/jolt/compiler.janet` | `analyze-form`, `emit*` dispatch, `compile-form`, symbol classifier | -| `src/jolt/loader.janet` | `load-ns`, `reload-ns`, in-memory bytecode cache | -| `test/compiler-test.janet` | Round-trip: compile-form → Janet eval → assert | - -Modified files: -- `evaluator.janet` — add compiler fast-path for `def`/`defn`/`defmacro` when `:compile?` set -- `types.janet` — add `:compiled-cache` table to context -- `api.janet` — expose `compile-string`, `load-ns`, `compile-file`; `init` gets `:compile?` flag -- `reader.janet` — **no change** -- `core.janet` — **no change** - -## Emit target advantages - -Both input and output are parenthesized prefix syntax, so many forms pass through almost unchanged: +Janet's `eval` does NOT have access to `use`-imported symbols from the calling file. `(eval "(core-inc 1)")` fails with "unknown symbol core-inc". The fix: emit Janet tuples where function VALUES are embedded: `[core-inc 1]`. ``` -Clojure: (defn f [x] (+ x 1)) -AST: {:op :def :name "f" :init {:op :fn :methods [...]}} -Janet: (defn f [x] (+ x 1)) ← nearly identical +core-fn-values table: "core-inc" → core-inc (the actual function) +emit-core-symbol-expr → (get core-fn-values janet-name) ``` -Main work: -- Symbol resolution (Clojure's `clojure.core/+` → Janet's `core-+`) -- Truthiness wrapping (`nil`/`false` are falsey in Clojure, Janet only `nil`) -- Special form mapping (`loop*`/`recur` → Janet `loop` + explicit recur vars) -- Vars → Janet table lookups +Source-to-source (`compile-form` + `emit-ast`) still exists for debugging but is NOT used by `compile-and-eval`. -## Implementation phases +## Macro expansion -| Phase | What | Deliverable | -|-------|------|-------------| -| 1 | `compiler.janet` — `analyze-form` skeleton + `emit*` for `const`, `do`, `if`, `let`, `fn`, `def`, `invoke` | Basic forms compile and run | -| 2 | Symbol classifier — resolve locals/vars/core at analyze time | No runtime `resolve-sym` in compiled code | -| 3 | `loader.janet` + `api.janet` wiring — `:compile?` flag, `load-ns`, caching | File-based namespace loading works | -| 4 | Macro integration — expand at analyze time via interpreter | Macros work in compiled code | -| 5 | Remaining ops: `loop`/`recur`, `try`/`throw`, `quote`, `syntax-quote`, `set!`, `deftype`, `.` | Full language coverage | -| 6 | Tests + benchmarks | Correctness + speedup measurement | +`analyze-form` checks whether the head symbol of a list resolves to a macro var before dispatching to special form handling: -## Pitfalls +1. Look up symbol in current ns → core ns via `resolve-macro` +2. If `var-macro?` is true, call `(var-get macro-var)` to get the fn +3. `(apply macro-fn (tuple/slice form 1))` to expand +4. `(analyze-form expanded ...)` to re-analyze the result -- Janet `compile` produces bytecode tied to Janet version — source-to-source avoids this -- CLJS analyzer is ~5000 lines; Jolt's can be simpler because emit target is s-expressions -- Symbol resolution must happen at analyze time, not runtime, for compiled code -- Macro expansion still uses interpreter at analyze time — macros are not AOT-compiled +Macros expand at analyze time, before emission. `defn` expands to `(def name (fn* ...))`, `when` to `(if test (do ...) nil)`, etc. + +## Symbol classification (in analyze-form) + +Order: qualified ns → local binding → core-symbol → bare symbol + +``` +(if (form :ns) → :qualified-symbol + (get bindings name) → :local + (get core-renames name) → :core-symbol + → :symbol) +``` + +core-renames MUST match actual fn names: `"-"` → `"core-sub"` (not `"core--"`), `"not"` → `"core-not"`. Verify against `core.janet` bindings. + +## core-fn-values + +Maps Janet string names to actual function values. Must be kept in sync with core-renames. When adding a new core fn, update BOTH tables. + +Functions that need special mapping (name differs): +- `"apply"` → `apply` (Janet built-in) +- `"-"` → `"core-sub"` (not `core--`) +- `"some"` → `core-some?` (shared with `core-some?`) +- `"pr-str"` → `core-str` (alias) +- `"nth"` → `core-get` (alias) + +## Stateful forms (must use interpreter, NOT compiler) + +These forms modify context state and cannot be compiled: +- `defmacro`, `ns`, `deftype`, `defmulti`, `defmethod`, `require`, `in-ns` + +Note: `def` IS handled by the compiler (compiles to Janet `def`). + +## Adding a new op + +1. Add `analyze-form` match arm for the special form +2. Add `emit-ast` match arm (source string path) +3. Add `emit-expr` match arm (data structure path) +4. Add `core-renames` entry if it's a core fn (name → Janet string name) +5. Add `core-fn-values` entry (Janet string name → actual fn value) +6. Add tests in `test/compiler-test.janet` + +## Test patterns + +- Source output tests: `(assert (= "(expected)" (compile-str "(input)")) "label")` +- Round-trip tests: `(assert (= val (compile-eval-str "(input)")) "label")` +- Compile flag tests: `(eval-string ctx "(input)")` with `{:compile? true}` + +Run: `janet test/compiler-test.janet` or `jpm test` diff --git a/.dirge/skills/jolt-dev/SKILL.md b/.dirge/skills/jolt-dev/SKILL.md index fbf049f..8eb08c6 100644 --- a/.dirge/skills/jolt-dev/SKILL.md +++ b/.dirge/skills/jolt-dev/SKILL.md @@ -37,9 +37,16 @@ For evaluating Janet source strings, use the parser pipeline: When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants. +## Compiler (see also `jolt-compiler` skill) + +`src/jolt/compiler.janet` — Clojure→Janet source compiler with macro expansion. +`test/compiler-test.janet` — 11 test groups covering all ops. + +Key design decision: **compile-and-eval emits Janet DATA STRUCTURES, not source strings**, because Janet's `eval` doesn't see `use`-imported symbols. `core-fn-values` table resolves Janet names to actual function values at compile time. + ## Special Form Checklist -To add a new special form to the evaluator: +To add a new special form to the evaluator AND compiler: 1. Add the name to `special-symbol?` in `src/jolt/evaluator.janet` 2. Add a match arm in `eval-list` (the match on `name`) diff --git a/src/jolt/api.janet b/src/jolt/api.janet index f9860f6..4483b3f 100644 --- a/src/jolt/api.janet +++ b/src/jolt/api.janet @@ -42,22 +42,24 @@ (defn eval-string "Evaluate a Clojure source string in a Jolt context. - When :compile? is enabled, compiles to Janet source and evaluates via Janet. - Stateful forms (def, defmacro, ns, deftype) always use the interpreter." + When :compile? is enabled, compiles to Janet and evaluates. + Macros are expanded at compile time. + Context-modifying forms (ns, defmacro, deftype, require, in-ns, defmulti, defmethod) + always use the interpreter." [ctx s] (let [compile? (get (ctx :env) :compile?) form (parse-string s)] (if (and compile? (array? form)) - # Check if this is a stateful form that needs the interpreter (let [first-form (first form) head-name (if (and (struct? first-form) (= :symbol (first-form :jolt/type))) (first-form :name) - nil)] - (if (or (= head-name "def") (= head-name "defmacro") (= head-name "ns") - (= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod") - (= head-name "require") (= head-name "in-ns")) + nil) + stateful? (or (= head-name "defmacro") (= head-name "ns") + (= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod") + (= head-name "require") (= head-name "in-ns"))] + (if stateful? (eval-form ctx @{} form) - (compile-and-eval form))) + (compile-and-eval form ctx))) (eval-form ctx @{} form)))) (defn eval-string* diff --git a/src/jolt/compiler.janet b/src/jolt/compiler.janet index 4467bce..a250919 100644 --- a/src/jolt/compiler.janet +++ b/src/jolt/compiler.janet @@ -1,10 +1,15 @@ # Jolt Compiler # Source-to-source: Clojure forms → Janet source # Two-phase: analyze-form (classify) → emit-ast (generate) +# +# When ctx is passed to analyze-form, macros are expanded at analyze time. + +(use ./types) +(use ./core) (def- core-renames @{"+" "core-+" - "-" "core--" + "-" "core-sub" "*" "core-*" "/" "core-/" "inc" "core-inc" @@ -101,11 +106,125 @@ (= name "defmulti") (= name "defmethod") (= name "locking"))) # ============================================================ -# Analyzer +# Macro resolution +# ============================================================ + +(defn- resolve-macro + "Resolve a symbol struct to a macro var. Returns the var or nil." + [ctx sym-s] + (when ctx + (let [name (sym-s :name) + ns-sym (sym-s :ns)] + (if ns-sym + (let [target-ns (ctx-find-ns ctx ns-sym) + v (ns-find target-ns name)] + (if (and v (var-macro? v)) v)) + (let [current-ns-name (ctx-current-ns ctx) + current-ns (ctx-find-ns ctx current-ns-name) + v (ns-find current-ns name)] + (if v + (if (var-macro? v) v) + (let [core-ns (ctx-find-ns ctx "clojure.core") + cv (ns-find core-ns name)] + (if (and cv (var-macro? cv)) cv)))))))) + +# ============================================================ +# Core function value lookup — resolved at compile time +# ============================================================ + +(def- core-fn-values + (let [t @{}] + (put t "core-+" core-+) + (put t "core-sub" core-sub) + (put t "core-*" core-*) + (put t "core-/" core-/) + (put t "core-inc" core-inc) + (put t "core-dec" core-dec) + (put t "core-=" core-=) + (put t "core-not=" core-not=) + (put t "core-<" core-<) + (put t "core->" core->) + (put t "core-<=" core-<=) + (put t "core->=" core->=) + (put t "core-nil?" core-nil?) + (put t "core-not" core-not) + (put t "core-some?" core-some?) + (put t "core-string?" core-string?) + (put t "core-number?" core-number?) + (put t "core-fn?" core-fn?) + (put t "core-keyword?" core-keyword?) + (put t "core-symbol?" core-symbol?) + (put t "core-vector?" core-vector?) + (put t "core-map?" core-map?) + (put t "core-seq?" core-seq?) + (put t "core-coll?" core-coll?) + (put t "core-true?" core-true?) + (put t "core-false?" core-false?) + (put t "core-identical?" core-identical?) + (put t "core-zero?" core-zero?) + (put t "core-pos?" core-pos?) + (put t "core-neg?" core-neg?) + (put t "core-even?" core-even?) + (put t "core-odd?" core-odd?) + (put t "core-empty?" core-empty?) + (put t "core-every?" core-every?) + (put t "core-first" core-first) + (put t "core-rest" core-rest) + (put t "core-next" core-next) + (put t "core-cons" core-cons) + (put t "core-conj" core-conj) + (put t "core-assoc" core-assoc) + (put t "core-dissoc" core-dissoc) + (put t "core-get" core-get) + (put t "core-get-in" core-get-in) + (put t "core-contains?" core-contains?) + (put t "core-count" core-count) + (put t "core-seq" core-seq) + (put t "core-vec" core-vec) + (put t "core-map" core-map) + (put t "core-filter" core-filter) + (put t "core-remove" core-remove) + (put t "core-reduce" core-reduce) + (put t "core-str" core-str) + (put t "core-prn" core-prn) + (put t "core-println" core-println) + (put t "core-print" core-print) + (put t "core-identity" core-identity) + (put t "core-comp" core-comp) + (put t "core-partial" core-partial) + (put t "core-complement" core-complement) + (put t "core-constantly" core-constantly) + (put t "core-memoize" core-memoize) + (put t "core-range" core-range) + (put t "core-take" core-take) + (put t "core-drop" core-drop) + (put t "core-take-while" core-take-while) + (put t "core-drop-while" core-drop-while) + (put t "core-reverse" core-reverse) + (put t "core-into" core-into) + (put t "core-merge" core-merge) + (put t "core-merge-with" core-merge-with) + (put t "core-keys" core-keys) + (put t "core-vals" core-vals) + (put t "core-zipmap" core-zipmap) + (put t "core-select-keys" core-select-keys) + (put t "core-max" core-max) + (put t "core-min" core-min) + (put t "core-quot" core-quot) + (put t "core-rem" core-rem) + (put t "core-mod" core-mod) + (put t "core-apply" apply) + (put t "core-some" core-some?) + (put t "core-pr-str" core-str) + (put t "core-nth" core-get) + t)) # ============================================================ (defn analyze-form - [form bindings] + "Analyze a Clojure form and return an AST node with :op key. + Takes bindings (table) and optional ctx (for macro expansion)." + [form bindings &opt ctx] + (default ctx nil) (cond (literal? form) {:op :const :val form} @@ -126,78 +245,87 @@ head-name (if (and (struct? first-form) (= :symbol (first-form :jolt/type))) (first-form :name) nil)] - (if head-name - (match head-name - "quote" {:op :quote :expr (in form 1)} - "do" (let [all-statements (array/slice form 1) - n (length all-statements) - analyzed (map |(analyze-form $ bindings) all-statements)] - {:op :do - :statements (array/slice analyzed 0 (- n 1)) - :ret (in analyzed (- n 1))}) - "if" {:op :if - :test (analyze-form (in form 1) bindings) - :then (analyze-form (in form 2) bindings) - :else (if (> (length form) 3) - (analyze-form (in form 3) bindings) - {:op :const :val nil})} - "def" {:op :def - :name (in form 1) - :init (analyze-form (in form 2) bindings)} - "fn*" (let [params (in form 1) - body-bindings (do - (var bb @{}) - (loop [[k v] :pairs bindings] (put bb k v)) - (each p params - (put bb (if (struct? p) (p :name) p) :jolt/local)) - bb) - body-exprs (tuple/slice form 2) - analyzed-body (map |(analyze-form $ body-bindings) body-exprs) - n-body (length analyzed-body)] - {:op :fn :params params - :body (if (> n-body 1) - {:op :do - :statements (array/slice analyzed-body 0 (- n-body 1)) - :ret (last analyzed-body)} - (first analyzed-body))}) - "let*" (let [bind-vec (in form 1) - body-exprs (tuple/slice form 2) - binding-pairs (do - (var pairs @[]) - (var i 0) - (let [n (length bind-vec)] - (while (< i n) - (let [sym-s (in bind-vec i) - name (if (struct? sym-s) (sym-s :name) sym-s) - val-form (if (< (+ i 1) n) (in bind-vec (+ i 1)) nil) - val-ast (if val-form (analyze-form val-form bindings) {:op :const :val nil})] - (array/push pairs {:name name :init val-ast}) - (+= i 2)))) - pairs) - body-bindings (do - (var bb @{}) - (loop [[k v] :pairs bindings] (put bb k v)) - (each bp binding-pairs - (put bb (bp :name) :jolt/local)) - bb) - analyzed-body (map |(analyze-form $ body-bindings) body-exprs) - n-body (length analyzed-body)] - {:op :let - :binding-pairs binding-pairs - :body (if (> n-body 1) - {:op :do - :statements (array/slice analyzed-body 0 (- n-body 1)) - :ret (last analyzed-body)} - (first analyzed-body))}) - (let [f-ast (analyze-form first-form bindings) - args (map |(analyze-form $ bindings) (tuple/slice form 1))] - {:op :invoke :fn f-ast :args args})) - (let [f-ast (analyze-form first-form bindings) - args (map |(analyze-form $ bindings) (tuple/slice form 1))] - {:op :invoke :fn f-ast :args args}))) + # Macro expansion: if ctx is provided and head resolves to a macro, + # expand it and re-analyze the expanded form + (if (and ctx head-name + (not (special-form? head-name)) + (resolve-macro ctx first-form)) + (let [macro-var (resolve-macro ctx first-form) + macro-fn (var-get macro-var) + expanded (apply macro-fn (tuple/slice form 1))] + (analyze-form expanded bindings ctx)) + (if head-name + (match head-name + "quote" {:op :quote :expr (in form 1)} + "do" (let [all-statements (array/slice form 1) + n (length all-statements) + analyzed (map |(analyze-form $ bindings ctx) all-statements)] + {:op :do + :statements (array/slice analyzed 0 (- n 1)) + :ret (in analyzed (- n 1))}) + "if" {:op :if + :test (analyze-form (in form 1) bindings ctx) + :then (analyze-form (in form 2) bindings ctx) + :else (if (> (length form) 3) + (analyze-form (in form 3) bindings ctx) + {:op :const :val nil})} + "def" {:op :def + :name (in form 1) + :init (analyze-form (in form 2) bindings ctx)} + "fn*" (let [params (in form 1) + body-bindings (do + (var bb @{}) + (loop [[k v] :pairs bindings] (put bb k v)) + (each p params + (put bb (if (struct? p) (p :name) p) :jolt/local)) + bb) + body-exprs (tuple/slice form 2) + analyzed-body (map |(analyze-form $ body-bindings ctx) body-exprs) + n-body (length analyzed-body)] + {:op :fn :params params + :body (if (> n-body 1) + {:op :do + :statements (array/slice analyzed-body 0 (- n-body 1)) + :ret (last analyzed-body)} + (first analyzed-body))}) + "let*" (let [bind-vec (in form 1) + body-exprs (tuple/slice form 2) + binding-pairs (do + (var pairs @[]) + (var i 0) + (let [n (length bind-vec)] + (while (< i n) + (let [sym-s (in bind-vec i) + name (if (struct? sym-s) (sym-s :name) sym-s) + val-form (if (< (+ i 1) n) (in bind-vec (+ i 1)) nil) + val-ast (if val-form (analyze-form val-form bindings ctx) {:op :const :val nil})] + (array/push pairs {:name name :init val-ast}) + (+= i 2)))) + pairs) + body-bindings (do + (var bb @{}) + (loop [[k v] :pairs bindings] (put bb k v)) + (each bp binding-pairs + (put bb (bp :name) :jolt/local)) + bb) + analyzed-body (map |(analyze-form $ body-bindings ctx) body-exprs) + n-body (length analyzed-body)] + {:op :let + :binding-pairs binding-pairs + :body (if (> n-body 1) + {:op :do + :statements (array/slice analyzed-body 0 (- n-body 1)) + :ret (last analyzed-body)} + (first analyzed-body))}) + (let [f-ast (analyze-form first-form bindings ctx) + args (map |(analyze-form $ bindings ctx) (tuple/slice form 1))] + {:op :invoke :fn f-ast :args args})) + (let [f-ast (analyze-form first-form bindings ctx) + args (map |(analyze-form $ bindings ctx) (tuple/slice form 1))] + {:op :invoke :fn f-ast :args args})))) (tuple? form) - (let [items (map |(analyze-form $ bindings) form)] + (let [items (map |(analyze-form $ bindings ctx) form)] {:op :vector :items items}) (struct? form) @@ -211,8 +339,7 @@ (var emit-ast nil) -(defn- emit-const-str - [val buf] +(defn- emit-const-str [val buf] (cond (nil? val) (buffer/push buf "nil") (= true val) (buffer/push buf "true") @@ -313,30 +440,110 @@ :quote (emit-quote-str (ast :expr) buf) (buffer/push buf (string "/* unhandled op: " (ast :op) " */"))))) +# ============================================================ +# Emitter — AST → Janet data structure (for direct eval) +# ============================================================ + +(var emit-expr nil) + +(defn- emit-const-expr [val] val) + +(defn- emit-do-expr [statements ret] + (def exprs @['do]) + (each s statements (array/push exprs (emit-expr s))) + (when ret (array/push exprs (emit-expr ret))) + (tuple/slice (tuple ;exprs))) + +(defn- emit-if-expr [test then else] + (def exprs @['if]) + (array/push exprs (emit-expr test)) + (array/push exprs (emit-expr then)) + (when else (array/push exprs (emit-expr else))) + (tuple/slice (tuple ;exprs))) + +(defn- emit-def-expr [name-sym init] + ['def (symbol (name-sym :name)) (emit-expr init)]) + +(defn- emit-fn-expr [params body] + (def param-syms @[]) + (each p params + (array/push param-syms (symbol (if (struct? p) (p :name) p)))) + ['fn (tuple/slice (tuple ;param-syms)) (emit-expr body)]) + +(defn- emit-let-expr [binding-pairs body] + (def bind-tuple @[]) + (each bp binding-pairs + (array/push bind-tuple (symbol (bp :name))) + (array/push bind-tuple (emit-expr (bp :init)))) + ['let (tuple/slice (tuple ;bind-tuple)) (emit-expr body)]) + +(defn- emit-invoke-expr [f-ast args] + (def exprs @[(emit-expr f-ast)]) + (each arg args (array/push exprs (emit-expr arg))) + (tuple/slice (tuple ;exprs))) + +(defn- emit-symbol-expr [name] (symbol name)) +(defn- emit-local-expr [name] (symbol name)) + +(defn- emit-core-symbol-expr [janet-name] + (or (get core-fn-values janet-name) + (error (string "Core fn not found: " janet-name)))) + +(defn- emit-qualified-symbol-expr [ns name] + (error (string "Cannot eval qualified symbol at compile time: " ns "/" name))) + +(defn- emit-vector-expr [items] + (def exprs @[]) + (each item items (array/push exprs (emit-expr item))) + (tuple/slice (tuple ;exprs))) + +(defn- emit-map-expr [form] form) + +(defn- emit-quote-expr [expr] + ['quote (analyze-form expr @{})]) + +(set emit-expr + (fn [ast] + (match (ast :op) + :const (emit-const-expr (ast :val)) + :symbol (emit-symbol-expr (ast :name)) + :local (emit-local-expr (ast :name)) + :core-symbol (emit-core-symbol-expr (ast :janet-name)) + :qualified-symbol (emit-qualified-symbol-expr (ast :ns) (ast :name)) + :do (emit-do-expr (ast :statements) (ast :ret)) + :if (emit-if-expr (ast :test) (ast :then) (ast :else)) + :def (emit-def-expr (ast :name) (ast :init)) + :fn (emit-fn-expr (ast :params) (ast :body)) + :let (emit-let-expr (ast :binding-pairs) (ast :body)) + :invoke (emit-invoke-expr (ast :fn) (ast :args)) + :vector (emit-vector-expr (ast :items)) + :map (emit-map-expr (ast :form)) + :quote (emit-quote-expr (ast :expr)) + (error (string "Unhandled op: " (ast :op)))))) + # ============================================================ # Public API # ============================================================ (defn compile-form - "Compile a Clojure form to a Janet source string." - [form] - (let [ast (analyze-form form @{}) + "Compile a Clojure form to a Janet source string. + Pass ctx for macro expansion." + [form &opt ctx] + (default ctx nil) + (let [ast (analyze-form form @{} ctx) buf @""] (emit-ast ast buf) (string buf))) -(defn eval-janet-source - "Parse and evaluate a Janet source string. - Uses the proper parser→produce→eval pipeline so special forms work." - [source] - (def p (parser/new)) - (parser/consume p source) - (parser/eof p) - (def form (parser/produce p)) - (eval form)) +(defn compile-ast + "Compile a Clojure form to an eval-able Janet data structure. + Core function symbols are resolved to actual function values." + [form &opt ctx] + (default ctx nil) + (emit-expr (analyze-form form @{} ctx))) (defn compile-and-eval - "Compile a Clojure form to Janet source and evaluate it. - Returns the result value." - [form] - (eval-janet-source (compile-form form))) + "Compile a Clojure form and evaluate it as Janet. + Emits Janet data structures with resolved core functions." + [form ctx] + (eval (compile-ast form ctx))) diff --git a/src/jolt/loader.janet b/src/jolt/loader.janet index 4a2d09a..b2197a3 100644 --- a/src/jolt/loader.janet +++ b/src/jolt/loader.janet @@ -50,9 +50,8 @@ (put cache ns-name cached)) (each form forms - (let [janet-src (compile-form form)] - (array/push cached janet-src) - (eval-janet-source janet-src))) + (array/push cached form) + (compile-and-eval form ctx)) ns-name) # Interpreter path (do diff --git a/test/compiler-test.janet b/test/compiler-test.janet index 034ae05..c02610f 100644 --- a/test/compiler-test.janet +++ b/test/compiler-test.janet @@ -102,7 +102,7 @@ (defn compile-eval-str [s] (let [form (parse-string s)] - (compile-and-eval form))) + (compile-and-eval form nil))) (assert (= 42 (compile-eval-str "42")) "eval literal") (assert (= 2 (compile-eval-str "(inc 1)")) "eval inc") @@ -141,3 +141,32 @@ (print " passed") (print "\nAll compiler Phase 3 tests passed!") + +# ============================================================ +# 11. Macro expansion (Phase 4) +# ============================================================ +(print "11: macro expansion...") +(use ../src/jolt/api) + +(let [ctx (init {:compile? true})] + # defn expands via compiler, produces Janet def + (eval-string ctx "(defn square [n] (* n n))") + (assert (= 25 (eval-string ctx "(square 5)")) "defn via compiler") + + # when macro + (assert (= 42 (eval-string ctx "(when true 42)")) "when true") + (assert (= nil (eval-string ctx "(when false 42)")) "when false") + + # let macro + (assert (= 30 (eval-string ctx "(let [x 10 y 20] (+ x y))")) "let macro") + + # fn macro + (assert (= 49 (eval-string ctx "((fn [x] (* x x)) 7)")) "fn macro") + + # and/or + (assert (= 3 (eval-string ctx "(and 1 2 3)")) "and") + (assert (= 99 (eval-string ctx "(or nil false 99)")) "or")) + +(print " passed") + +(print "\nAll compiler Phase 4 tests passed!")