Phase 4: Macro expansion in compiler

- resolve-macro: resolve symbols to macro vars via ctx
- Macro expansion in analyze-form: detects macro heads, expands, re-analyzes
- compile-ast: emits Janet data structures with resolved core fn values
- compile-and-eval uses compile-ast (no source parse roundtrip)
- eval-string routes macros through compiler (expanded at analyze time)
- Fix - mapping: core-sub (core-- doesn't exist)
- All 317 tests pass + 6 new Phase 4 macro tests
This commit is contained in:
Yogthos 2026-06-02 15:43:08 -04:00
parent ab7ff85816
commit a8c453183f
9 changed files with 448 additions and 188 deletions

View file

@ -1,5 +1,5 @@
Compiler Phases 1-3 COMPLETE. src/jolt/compiler.janet (342 lines): analyze-form→emit-ast two-phase. Phase 2: symbol classification (locals→core→symbol, shadowing works). Phase 3: compile-and-eval via parser→consume→eof→produce→eval pipeline, :compile? flag in types.janet context, loader.janet (80 lines) for namespace loading, api.janet updated (compile-string, compile-file, compile-aware eval-string). Stateful forms (def, defmacro, ns, deftype, defmulti, defmethod, require, in-ns) always route to interpreter. All 317 tests pass. Next: Phase 4 (macro integration), Phase 5 (remaining ops: loop/recur, try/throw, quote, syntax-quote, set!, .), Phase 6 (benchmarks).
Compiler in `src/jolt/compiler.janet` (~470 lines). Three emit modes: 1) compile-form → Janet source string (debug/display) 2) compile-ast → Janet data structures with resolved fn values (for eval) 3) compile-and-eval → compile-ast + eval. core-renames maps Clojure→Janet STRING names. core-fn-values maps Janet string names→actual function VALUES (used by compile-ast). analyze-form [form bindings ctx] — ctx optional for macro expansion. Macro expansion: when head symbol resolves to macro var, apply macro fn to args and re-analyze expanded form. Ops: const, do, if, def, fn*, let*, invoke, symbol (local/core/qualified), quote, vector, map. Symbol classification: locals → core-symbol → symbol (shadowing works). compile-form and compile-ast accept optional ctx arg.
§
Compiler is in `src/jolt/compiler.janet` (342 lines). Two-phase: `analyze-form [form bindings]` → annotated AST with `:op` keys, `emit-ast` dispatches on `:op` → Janet source string via StringBuffer. `compile-form` for source output, `compile-and-eval` for direct eval via `eval-janet-source` (parser/new→consume→eof→produce→eval pipeline). `core-renames` table maps Clojure→Janet names. Ops covered: const, do, if, def, fn*, let*, invoke, symbol (local/core-symbol/qualified classification), quote. Symbol classification order: locals → core-symbol → symbol (shadowing works). Stateful forms (def, defmacro, etc.) must use interpreter path, not compile-and-eval.
Compiler tests in `test/compiler-test.janet` (172 lines, 11 test groups). Groups: 1-literals, 2-do, 3-if, 4-def, 5-fn, 6-let, 7-invoke, 8-local classification, 9-compile-and-eval round-trip, 10-compile flag, 11-macro expansion (defn, when, let, fn, and/or via compile?). All 317 tests pass.
§
Compiler tests in `test/compiler-test.janet` (143 lines, 10 test groups). Phase 1: 7 groups, 19 string-output assertions. Phase 2: local classification (3 assertions: shadowing, param shadowing, nested let). Phase 3: compile-and-eval round-trip (9 assertions) + :compile? flag tests (7 assertions). All 317 tests pass. Run with `janet test/compiler-test.janet` or `jpm test`.
Janet's `eval` runs in Janet's default environment and does NOT have access to symbols imported via `(use ...)` in the calling file. `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. FIX: emit Janet data structures where function VALUES are embedded directly (e.g. `[core-inc 1]`) rather than source strings `"(core-inc 1)"`. The `core-fn-values` table resolves Janet symbol names to actual function values at compile time.

View file

@ -1,5 +1,5 @@
Janet LSP produces false positives on `.janet` files — it doesn't understand Janet syntax (thinks docstring lines are unresolved symbols, doesn't know `declare-project`/`declare-source` macros, etc.). These are pre-existing and should be ignored — they don't affect runtime correctness.
§
Janet `(parse s)` returns `[symbol, error-position]` for forms — e.g., `(parse "(+ 1 2)")` gives `[+ 1]`, not a parsed tuple. For proper evaluation of special forms, must use the parser pipeline: `parser/new``parser/consume``parser/eof``parser/produce``eval`. Without `parser/eof`, `produce` returns nil for simple forms like numbers. Janet's `eval` also doesn't support special forms natively in data structures — `(eval [if true 1 2])` fails with "unknown symbol if". Must parse strings, not eval tuples.
Janet eval scope & source-to-source failure: `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. Janet's `eval` runs in the default environment and doesn't see `use`-imported symbols. NEVER emit Janet source strings for eval. Instead emit Janet data structures where function VALUES are embedded directly: `[core-inc 1]` not `"(core-inc 1)"`. Use a `core-fn-values` table to resolve names at compile time. The `(parse s)` function returns `[symbol, error-position]` — must use parser pipeline for proper eval: `parser/new→consume→eof→produce→eval`. Without `parser/eof`, numbers/literals return nil from produce. Janet's `eval` also doesn't support special forms natively in data structures — `(eval [if true 1 2])` fails with "unknown symbol if". Must parse strings, not eval tuples.
§
When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants — `(set ns-name ...)` on a `def` fails with "cannot set constant". This hit us in loader.janet ns-name extraction loop.
§
core-renames MUST match actual function names defined in core.janet. `"-"` maps to `core-sub` NOT `core--`. `"not"` maps to `core-not` (defined as `(defn core-not ...)`). Missing entries cause silent nil returns. When adding to core-renames, grep core.janet for the actual `(defn core-XXX ...)` or `(def core-XXX ...)` name. Then add matching entry to core-fn-values.