Phase 4: Macro expansion in compiler
- resolve-macro: resolve symbols to macro vars via ctx - Macro expansion in analyze-form: detects macro heads, expands, re-analyzes - compile-ast: emits Janet data structures with resolved core fn values - compile-and-eval uses compile-ast (no source parse roundtrip) - eval-string routes macros through compiler (expanded at analyze time) - Fix - mapping: core-sub (core-- doesn't exist) - All 317 tests pass + 6 new Phase 4 macro tests
This commit is contained in:
Yogthos
parent
ab7ff85816
commit
a8c453183f
9 changed files with 448 additions and 188 deletions
|
|
@ -1,5 +1,5 @@
|
|||
Compiler Phases 1-3 COMPLETE. src/jolt/compiler.janet (342 lines): analyze-form→emit-ast two-phase. Phase 2: symbol classification (locals→core→symbol, shadowing works). Phase 3: compile-and-eval via parser→consume→eof→produce→eval pipeline, :compile? flag in types.janet context, loader.janet (80 lines) for namespace loading, api.janet updated (compile-string, compile-file, compile-aware eval-string). Stateful forms (def, defmacro, ns, deftype, defmulti, defmethod, require, in-ns) always route to interpreter. All 317 tests pass. Next: Phase 4 (macro integration), Phase 5 (remaining ops: loop/recur, try/throw, quote, syntax-quote, set!, .), Phase 6 (benchmarks).
|
||||
Compiler in `src/jolt/compiler.janet` (~470 lines). Three emit modes: 1) compile-form → Janet source string (debug/display) 2) compile-ast → Janet data structures with resolved fn values (for eval) 3) compile-and-eval → compile-ast + eval. core-renames maps Clojure→Janet STRING names. core-fn-values maps Janet string names→actual function VALUES (used by compile-ast). analyze-form [form bindings ctx] — ctx optional for macro expansion. Macro expansion: when head symbol resolves to macro var, apply macro fn to args and re-analyze expanded form. Ops: const, do, if, def, fn*, let*, invoke, symbol (local/core/qualified), quote, vector, map. Symbol classification: locals → core-symbol → symbol (shadowing works). compile-form and compile-ast accept optional ctx arg.
|
||||
§
|
||||
Compiler is in `src/jolt/compiler.janet` (342 lines). Two-phase: `analyze-form [form bindings]` → annotated AST with `:op` keys, `emit-ast` dispatches on `:op` → Janet source string via StringBuffer. `compile-form` for source output, `compile-and-eval` for direct eval via `eval-janet-source` (parser/new→consume→eof→produce→eval pipeline). `core-renames` table maps Clojure→Janet names. Ops covered: const, do, if, def, fn*, let*, invoke, symbol (local/core-symbol/qualified classification), quote. Symbol classification order: locals → core-symbol → symbol (shadowing works). Stateful forms (def, defmacro, etc.) must use interpreter path, not compile-and-eval.
|
||||
Compiler tests in `test/compiler-test.janet` (172 lines, 11 test groups). Groups: 1-literals, 2-do, 3-if, 4-def, 5-fn, 6-let, 7-invoke, 8-local classification, 9-compile-and-eval round-trip, 10-compile flag, 11-macro expansion (defn, when, let, fn, and/or via compile?). All 317 tests pass.
|
||||
§
|
||||
Compiler tests in `test/compiler-test.janet` (143 lines, 10 test groups). Phase 1: 7 groups, 19 string-output assertions. Phase 2: local classification (3 assertions: shadowing, param shadowing, nested let). Phase 3: compile-and-eval round-trip (9 assertions) + :compile? flag tests (7 assertions). All 317 tests pass. Run with `janet test/compiler-test.janet` or `jpm test`.
|
||||
Janet's `eval` runs in Janet's default environment and does NOT have access to symbols imported via `(use ...)` in the calling file. `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. FIX: emit Janet data structures where function VALUES are embedded directly (e.g. `[core-inc 1]`) rather than source strings `"(core-inc 1)"`. The `core-fn-values` table resolves Janet symbol names to actual function values at compile time.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
Janet LSP produces false positives on `.janet` files — it doesn't understand Janet syntax (thinks docstring lines are unresolved symbols, doesn't know `declare-project`/`declare-source` macros, etc.). These are pre-existing and should be ignored — they don't affect runtime correctness.
|
||||
§
|
||||
Janet `(parse s)` returns `[symbol, error-position]` for forms — e.g., `(parse "(+ 1 2)")` gives `[+ 1]`, not a parsed tuple. For proper evaluation of special forms, must use the parser pipeline: `parser/new` → `parser/consume` → `parser/eof` → `parser/produce` → `eval`. Without `parser/eof`, `produce` returns nil for simple forms like numbers. Janet's `eval` also doesn't support special forms natively in data structures — `(eval [if true 1 2])` fails with "unknown symbol if". Must parse strings, not eval tuples.
|
||||
Janet eval scope & source-to-source failure: `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. Janet's `eval` runs in the default environment and doesn't see `use`-imported symbols. NEVER emit Janet source strings for eval. Instead emit Janet data structures where function VALUES are embedded directly: `[core-inc 1]` not `"(core-inc 1)"`. Use a `core-fn-values` table to resolve names at compile time. The `(parse s)` function returns `[symbol, error-position]` — must use parser pipeline for proper eval: `parser/new→consume→eof→produce→eval`. Without `parser/eof`, numbers/literals return nil from produce. Janet's `eval` also doesn't support special forms natively in data structures — `(eval [if true 1 2])` fails with "unknown symbol if". Must parse strings, not eval tuples.
|
||||
§
|
||||
When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants — `(set ns-name ...)` on a `def` fails with "cannot set constant". This hit us in loader.janet ns-name extraction loop.
|
||||
§
|
||||
core-renames MUST match actual function names defined in core.janet. `"-"` maps to `core-sub` NOT `core--`. `"not"` maps to `core-not` (defined as `(defn core-not ...)`). Missing entries cause silent nil returns. When adding to core-renames, grep core.janet for the actual `(defn core-XXX ...)` or `(def core-XXX ...)` name. Then add matching entry to core-fn-values.
|
||||
|
|
|
|||
|
|
@ -1,22 +1,12 @@
|
|||
{
|
||||
"jpm-build": {
|
||||
"created_by": "agent",
|
||||
"use_count": 0,
|
||||
"view_count": 11,
|
||||
"patch_count": 0,
|
||||
"last_viewed_at": "2026-06-02T17:43:38.327259+00:00",
|
||||
"created_at": "2026-06-01T20:56:39.144222+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
},
|
||||
"jolt-dev": {
|
||||
"created_by": "agent",
|
||||
"use_count": 25,
|
||||
"view_count": 36,
|
||||
"patch_count": 31,
|
||||
"last_used_at": "2026-06-02T19:13:37.620054+00:00",
|
||||
"last_viewed_at": "2026-06-02T19:13:37.612317+00:00",
|
||||
"last_patched_at": "2026-06-02T19:14:06.338745+00:00",
|
||||
"use_count": 26,
|
||||
"view_count": 37,
|
||||
"patch_count": 32,
|
||||
"last_used_at": "2026-06-02T19:38:51.833410+00:00",
|
||||
"last_viewed_at": "2026-06-02T19:38:51.825602+00:00",
|
||||
"last_patched_at": "2026-06-02T19:39:44.919664+00:00",
|
||||
"created_at": "2026-06-01T21:26:06.614465+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
|
|
@ -25,7 +15,8 @@
|
|||
"created_by": "agent",
|
||||
"use_count": 0,
|
||||
"view_count": 0,
|
||||
"patch_count": 0,
|
||||
"patch_count": 1,
|
||||
"last_patched_at": "2026-06-02T19:40:03.526270+00:00",
|
||||
"created_at": "2026-06-02T17:54:38.690279+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
|
|
@ -41,5 +32,15 @@
|
|||
"created_at": "2026-06-01T21:49:51.101718+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
},
|
||||
"jpm-build": {
|
||||
"created_by": "agent",
|
||||
"use_count": 0,
|
||||
"view_count": 11,
|
||||
"patch_count": 0,
|
||||
"last_viewed_at": "2026-06-02T17:43:38.327259+00:00",
|
||||
"created_at": "2026-06-01T20:56:39.144222+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
}
|
||||
}
|
||||
|
|
@ -1,82 +1,97 @@
|
|||
---
|
||||
description: Jolt compiler architecture and implementation plan
|
||||
triggers:
|
||||
- "compile jolt"
|
||||
- "jolt compiler"
|
||||
- "Clojure to Janet compilation"
|
||||
- "add new op to compiler"
|
||||
- "fix compiler"
|
||||
---
|
||||
|
||||
# Jolt Compiler
|
||||
|
||||
Two-phase source-to-source compiler: Clojure forms → annotated AST → Janet source → Janet bytecode.
|
||||
Source-to-source Clojure→Janet compiler. Two-phase: analyze-form (classify + macro expand) → emit-ast (generate).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Clojure source → Reader → raw AST
|
||||
↓
|
||||
analyze-form (classify symbols, produce :op AST)
|
||||
↓
|
||||
emit* dispatch (generate Janet source string)
|
||||
↓
|
||||
Janet compile → bytecode
|
||||
Clojure form → analyze-form [form bindings ctx] → AST {:op ...}
|
||||
↓ (if head = macro var)
|
||||
expand → re-analyze expanded form
|
||||
↓
|
||||
emit-ast (source string) or emit-expr (data structure)
|
||||
```
|
||||
|
||||
Follows CLJS `cljs.analyzer` / `cljs.compiler` pattern.
|
||||
Three public entry points:
|
||||
- `(compile-form form &opt ctx)` → Janet source string (debug/display)
|
||||
- `(compile-ast form &opt ctx)` → Janet data structure (for eval)
|
||||
- `(compile-and-eval form ctx)` → compile-ast + eval
|
||||
|
||||
## Key decisions
|
||||
## Why data structures, not source strings
|
||||
|
||||
- **Target**: Janet source text (fed to Janet's `compile`), not direct bytecode. Simpler, debuggable, portable across Janet versions.
|
||||
- **Mode gating**: Opt-in per context via `:compile?` flag on `init`. `eval-string` still interprets unless opted in.
|
||||
- **Caching**: In-memory bytecode cache in context first. Disk persistence (`.jimage` files) as follow-up.
|
||||
|
||||
## AST ops (CLJS subset + Jolt-specific)
|
||||
|
||||
Core: `const`, `var`, `local`, `binding`, `if`, `do`, `let`, `loop`, `recur`, `fn`, `fn-method`, `def`, `invoke`, `quote`, `try`, `throw`, `set!`, `new`, `host-field`, `host-call`
|
||||
|
||||
Jolt-specific: `deftype`, `defmulti`, `defmethod`, `syntax-quote`
|
||||
|
||||
## File layout
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `src/jolt/compiler.janet` | `analyze-form`, `emit*` dispatch, `compile-form`, symbol classifier |
|
||||
| `src/jolt/loader.janet` | `load-ns`, `reload-ns`, in-memory bytecode cache |
|
||||
| `test/compiler-test.janet` | Round-trip: compile-form → Janet eval → assert |
|
||||
|
||||
Modified files:
|
||||
- `evaluator.janet` — add compiler fast-path for `def`/`defn`/`defmacro` when `:compile?` set
|
||||
- `types.janet` — add `:compiled-cache` table to context
|
||||
- `api.janet` — expose `compile-string`, `load-ns`, `compile-file`; `init` gets `:compile?` flag
|
||||
- `reader.janet` — **no change**
|
||||
- `core.janet` — **no change**
|
||||
|
||||
## Emit target advantages
|
||||
|
||||
Both input and output are parenthesized prefix syntax, so many forms pass through almost unchanged:
|
||||
Janet's `eval` does NOT have access to `use`-imported symbols from the calling file. `(eval "(core-inc 1)")` fails with "unknown symbol core-inc". The fix: emit Janet tuples where function VALUES are embedded: `[core-inc 1]`.
|
||||
|
||||
```
|
||||
Clojure: (defn f [x] (+ x 1))
|
||||
AST: {:op :def :name "f" :init {:op :fn :methods [...]}}
|
||||
Janet: (defn f [x] (+ x 1)) ← nearly identical
|
||||
core-fn-values table: "core-inc" → core-inc (the actual function)
|
||||
emit-core-symbol-expr → (get core-fn-values janet-name)
|
||||
```
|
||||
|
||||
Main work:
|
||||
- Symbol resolution (Clojure's `clojure.core/+` → Janet's `core-+`)
|
||||
- Truthiness wrapping (`nil`/`false` are falsey in Clojure, Janet only `nil`)
|
||||
- Special form mapping (`loop*`/`recur` → Janet `loop` + explicit recur vars)
|
||||
- Vars → Janet table lookups
|
||||
Source-to-source (`compile-form` + `emit-ast`) still exists for debugging but is NOT used by `compile-and-eval`.
|
||||
|
||||
## Implementation phases
|
||||
## Macro expansion
|
||||
|
||||
| Phase | What | Deliverable |
|
||||
|-------|------|-------------|
|
||||
| 1 | `compiler.janet` — `analyze-form` skeleton + `emit*` for `const`, `do`, `if`, `let`, `fn`, `def`, `invoke` | Basic forms compile and run |
|
||||
| 2 | Symbol classifier — resolve locals/vars/core at analyze time | No runtime `resolve-sym` in compiled code |
|
||||
| 3 | `loader.janet` + `api.janet` wiring — `:compile?` flag, `load-ns`, caching | File-based namespace loading works |
|
||||
| 4 | Macro integration — expand at analyze time via interpreter | Macros work in compiled code |
|
||||
| 5 | Remaining ops: `loop`/`recur`, `try`/`throw`, `quote`, `syntax-quote`, `set!`, `deftype`, `.` | Full language coverage |
|
||||
| 6 | Tests + benchmarks | Correctness + speedup measurement |
|
||||
`analyze-form` checks whether the head symbol of a list resolves to a macro var before dispatching to special form handling:
|
||||
|
||||
## Pitfalls
|
||||
1. Look up symbol in current ns → core ns via `resolve-macro`
|
||||
2. If `var-macro?` is true, call `(var-get macro-var)` to get the fn
|
||||
3. `(apply macro-fn (tuple/slice form 1))` to expand
|
||||
4. `(analyze-form expanded ...)` to re-analyze the result
|
||||
|
||||
- Janet `compile` produces bytecode tied to Janet version — source-to-source avoids this
|
||||
- CLJS analyzer is ~5000 lines; Jolt's can be simpler because emit target is s-expressions
|
||||
- Symbol resolution must happen at analyze time, not runtime, for compiled code
|
||||
- Macro expansion still uses interpreter at analyze time — macros are not AOT-compiled
|
||||
Macros expand at analyze time, before emission. `defn` expands to `(def name (fn* ...))`, `when` to `(if test (do ...) nil)`, etc.
|
||||
|
||||
## Symbol classification (in analyze-form)
|
||||
|
||||
Order: qualified ns → local binding → core-symbol → bare symbol
|
||||
|
||||
```
|
||||
(if (form :ns) → :qualified-symbol
|
||||
(get bindings name) → :local
|
||||
(get core-renames name) → :core-symbol
|
||||
→ :symbol)
|
||||
```
|
||||
|
||||
core-renames MUST match actual fn names: `"-"` → `"core-sub"` (not `"core--"`), `"not"` → `"core-not"`. Verify against `core.janet` bindings.
|
||||
|
||||
## core-fn-values
|
||||
|
||||
Maps Janet string names to actual function values. Must be kept in sync with core-renames. When adding a new core fn, update BOTH tables.
|
||||
|
||||
Functions that need special mapping (name differs):
|
||||
- `"apply"` → `apply` (Janet built-in)
|
||||
- `"-"` → `"core-sub"` (not `core--`)
|
||||
- `"some"` → `core-some?` (shared with `core-some?`)
|
||||
- `"pr-str"` → `core-str` (alias)
|
||||
- `"nth"` → `core-get` (alias)
|
||||
|
||||
## Stateful forms (must use interpreter, NOT compiler)
|
||||
|
||||
These forms modify context state and cannot be compiled:
|
||||
- `defmacro`, `ns`, `deftype`, `defmulti`, `defmethod`, `require`, `in-ns`
|
||||
|
||||
Note: `def` IS handled by the compiler (compiles to Janet `def`).
|
||||
|
||||
## Adding a new op
|
||||
|
||||
1. Add `analyze-form` match arm for the special form
|
||||
2. Add `emit-ast` match arm (source string path)
|
||||
3. Add `emit-expr` match arm (data structure path)
|
||||
4. Add `core-renames` entry if it's a core fn (name → Janet string name)
|
||||
5. Add `core-fn-values` entry (Janet string name → actual fn value)
|
||||
6. Add tests in `test/compiler-test.janet`
|
||||
|
||||
## Test patterns
|
||||
|
||||
- Source output tests: `(assert (= "(expected)" (compile-str "(input)")) "label")`
|
||||
- Round-trip tests: `(assert (= val (compile-eval-str "(input)")) "label")`
|
||||
- Compile flag tests: `(eval-string ctx "(input)")` with `{:compile? true}`
|
||||
|
||||
Run: `janet test/compiler-test.janet` or `jpm test`
|
||||
|
|
|
|||
|
|
@ -37,9 +37,16 @@ For evaluating Janet source strings, use the parser pipeline:
|
|||
|
||||
When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants.
|
||||
|
||||
## Compiler (see also `jolt-compiler` skill)
|
||||
|
||||
`src/jolt/compiler.janet` — Clojure→Janet source compiler with macro expansion.
|
||||
`test/compiler-test.janet` — 11 test groups covering all ops.
|
||||
|
||||
Key design decision: **compile-and-eval emits Janet DATA STRUCTURES, not source strings**, because Janet's `eval` doesn't see `use`-imported symbols. `core-fn-values` table resolves Janet names to actual function values at compile time.
|
||||
|
||||
## Special Form Checklist
|
||||
|
||||
To add a new special form to the evaluator:
|
||||
To add a new special form to the evaluator AND compiler:
|
||||
|
||||
1. Add the name to `special-symbol?` in `src/jolt/evaluator.janet`
|
||||
2. Add a match arm in `eval-list` (the match on `name`)
|
||||
|
|
|
|||
|
|
@ -42,22 +42,24 @@
|
|||
|
||||
(defn eval-string
|
||||
"Evaluate a Clojure source string in a Jolt context.
|
||||
When :compile? is enabled, compiles to Janet source and evaluates via Janet.
|
||||
Stateful forms (def, defmacro, ns, deftype) always use the interpreter."
|
||||
When :compile? is enabled, compiles to Janet and evaluates.
|
||||
Macros are expanded at compile time.
|
||||
Context-modifying forms (ns, defmacro, deftype, require, in-ns, defmulti, defmethod)
|
||||
always use the interpreter."
|
||||
[ctx s]
|
||||
(let [compile? (get (ctx :env) :compile?)
|
||||
form (parse-string s)]
|
||||
(if (and compile? (array? form))
|
||||
# Check if this is a stateful form that needs the interpreter
|
||||
(let [first-form (first form)
|
||||
head-name (if (and (struct? first-form) (= :symbol (first-form :jolt/type)))
|
||||
(first-form :name)
|
||||
nil)]
|
||||
(if (or (= head-name "def") (= head-name "defmacro") (= head-name "ns")
|
||||
(= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod")
|
||||
(= head-name "require") (= head-name "in-ns"))
|
||||
nil)
|
||||
stateful? (or (= head-name "defmacro") (= head-name "ns")
|
||||
(= head-name "deftype") (= head-name "defmulti") (= head-name "defmethod")
|
||||
(= head-name "require") (= head-name "in-ns"))]
|
||||
(if stateful?
|
||||
(eval-form ctx @{} form)
|
||||
(compile-and-eval form)))
|
||||
(compile-and-eval form ctx)))
|
||||
(eval-form ctx @{} form))))
|
||||
|
||||
(defn eval-string*
|
||||
|
|
|
|||
|
|
@ -1,10 +1,15 @@
|
|||
# Jolt Compiler
|
||||
# Source-to-source: Clojure forms → Janet source
|
||||
# Two-phase: analyze-form (classify) → emit-ast (generate)
|
||||
#
|
||||
# When ctx is passed to analyze-form, macros are expanded at analyze time.
|
||||
|
||||
(use ./types)
|
||||
(use ./core)
|
||||
|
||||
(def- core-renames
|
||||
@{"+" "core-+"
|
||||
"-" "core--"
|
||||
"-" "core-sub"
|
||||
"*" "core-*"
|
||||
"/" "core-/"
|
||||
"inc" "core-inc"
|
||||
|
|
@ -101,11 +106,125 @@
|
|||
(= name "defmulti") (= name "defmethod") (= name "locking")))
|
||||
|
||||
# ============================================================
|
||||
# Analyzer
|
||||
# Macro resolution
|
||||
# ============================================================
|
||||
|
||||
(defn- resolve-macro
|
||||
"Resolve a symbol struct to a macro var. Returns the var or nil."
|
||||
[ctx sym-s]
|
||||
(when ctx
|
||||
(let [name (sym-s :name)
|
||||
ns-sym (sym-s :ns)]
|
||||
(if ns-sym
|
||||
(let [target-ns (ctx-find-ns ctx ns-sym)
|
||||
v (ns-find target-ns name)]
|
||||
(if (and v (var-macro? v)) v))
|
||||
(let [current-ns-name (ctx-current-ns ctx)
|
||||
current-ns (ctx-find-ns ctx current-ns-name)
|
||||
v (ns-find current-ns name)]
|
||||
(if v
|
||||
(if (var-macro? v) v)
|
||||
(let [core-ns (ctx-find-ns ctx "clojure.core")
|
||||
cv (ns-find core-ns name)]
|
||||
(if (and cv (var-macro? cv)) cv))))))))
|
||||
|
||||
# ============================================================
|
||||
# Core function value lookup — resolved at compile time
|
||||
# ============================================================
|
||||
|
||||
(def- core-fn-values
|
||||
(let [t @{}]
|
||||
(put t "core-+" core-+)
|
||||
(put t "core-sub" core-sub)
|
||||
(put t "core-*" core-*)
|
||||
(put t "core-/" core-/)
|
||||
(put t "core-inc" core-inc)
|
||||
(put t "core-dec" core-dec)
|
||||
(put t "core-=" core-=)
|
||||
(put t "core-not=" core-not=)
|
||||
(put t "core-<" core-<)
|
||||
(put t "core->" core->)
|
||||
(put t "core-<=" core-<=)
|
||||
(put t "core->=" core->=)
|
||||
(put t "core-nil?" core-nil?)
|
||||
(put t "core-not" core-not)
|
||||
(put t "core-some?" core-some?)
|
||||
(put t "core-string?" core-string?)
|
||||
(put t "core-number?" core-number?)
|
||||
(put t "core-fn?" core-fn?)
|
||||
(put t "core-keyword?" core-keyword?)
|
||||
(put t "core-symbol?" core-symbol?)
|
||||
(put t "core-vector?" core-vector?)
|
||||
(put t "core-map?" core-map?)
|
||||
(put t "core-seq?" core-seq?)
|
||||
(put t "core-coll?" core-coll?)
|
||||
(put t "core-true?" core-true?)
|
||||
(put t "core-false?" core-false?)
|
||||
(put t "core-identical?" core-identical?)
|
||||
(put t "core-zero?" core-zero?)
|
||||
(put t "core-pos?" core-pos?)
|
||||
(put t "core-neg?" core-neg?)
|
||||
(put t "core-even?" core-even?)
|
||||
(put t "core-odd?" core-odd?)
|
||||
(put t "core-empty?" core-empty?)
|
||||
(put t "core-every?" core-every?)
|
||||
(put t "core-first" core-first)
|
||||
(put t "core-rest" core-rest)
|
||||
(put t "core-next" core-next)
|
||||
(put t "core-cons" core-cons)
|
||||
(put t "core-conj" core-conj)
|
||||
(put t "core-assoc" core-assoc)
|
||||
(put t "core-dissoc" core-dissoc)
|
||||
(put t "core-get" core-get)
|
||||
(put t "core-get-in" core-get-in)
|
||||
(put t "core-contains?" core-contains?)
|
||||
(put t "core-count" core-count)
|
||||
(put t "core-seq" core-seq)
|
||||
(put t "core-vec" core-vec)
|
||||
(put t "core-map" core-map)
|
||||
(put t "core-filter" core-filter)
|
||||
(put t "core-remove" core-remove)
|
||||
(put t "core-reduce" core-reduce)
|
||||
(put t "core-str" core-str)
|
||||
(put t "core-prn" core-prn)
|
||||
(put t "core-println" core-println)
|
||||
(put t "core-print" core-print)
|
||||
(put t "core-identity" core-identity)
|
||||
(put t "core-comp" core-comp)
|
||||
(put t "core-partial" core-partial)
|
||||
(put t "core-complement" core-complement)
|
||||
(put t "core-constantly" core-constantly)
|
||||
(put t "core-memoize" core-memoize)
|
||||
(put t "core-range" core-range)
|
||||
(put t "core-take" core-take)
|
||||
(put t "core-drop" core-drop)
|
||||
(put t "core-take-while" core-take-while)
|
||||
(put t "core-drop-while" core-drop-while)
|
||||
(put t "core-reverse" core-reverse)
|
||||
(put t "core-into" core-into)
|
||||
(put t "core-merge" core-merge)
|
||||
(put t "core-merge-with" core-merge-with)
|
||||
(put t "core-keys" core-keys)
|
||||
(put t "core-vals" core-vals)
|
||||
(put t "core-zipmap" core-zipmap)
|
||||
(put t "core-select-keys" core-select-keys)
|
||||
(put t "core-max" core-max)
|
||||
(put t "core-min" core-min)
|
||||
(put t "core-quot" core-quot)
|
||||
(put t "core-rem" core-rem)
|
||||
(put t "core-mod" core-mod)
|
||||
(put t "core-apply" apply)
|
||||
(put t "core-some" core-some?)
|
||||
(put t "core-pr-str" core-str)
|
||||
(put t "core-nth" core-get)
|
||||
t))
|
||||
# ============================================================
|
||||
|
||||
(defn analyze-form
|
||||
[form bindings]
|
||||
"Analyze a Clojure form and return an AST node with :op key.
|
||||
Takes bindings (table) and optional ctx (for macro expansion)."
|
||||
[form bindings &opt ctx]
|
||||
(default ctx nil)
|
||||
(cond
|
||||
(literal? form)
|
||||
{:op :const :val form}
|
||||
|
|
@ -126,78 +245,87 @@
|
|||
head-name (if (and (struct? first-form) (= :symbol (first-form :jolt/type)))
|
||||
(first-form :name)
|
||||
nil)]
|
||||
(if head-name
|
||||
(match head-name
|
||||
"quote" {:op :quote :expr (in form 1)}
|
||||
"do" (let [all-statements (array/slice form 1)
|
||||
n (length all-statements)
|
||||
analyzed (map |(analyze-form $ bindings) all-statements)]
|
||||
{:op :do
|
||||
:statements (array/slice analyzed 0 (- n 1))
|
||||
:ret (in analyzed (- n 1))})
|
||||
"if" {:op :if
|
||||
:test (analyze-form (in form 1) bindings)
|
||||
:then (analyze-form (in form 2) bindings)
|
||||
:else (if (> (length form) 3)
|
||||
(analyze-form (in form 3) bindings)
|
||||
{:op :const :val nil})}
|
||||
"def" {:op :def
|
||||
:name (in form 1)
|
||||
:init (analyze-form (in form 2) bindings)}
|
||||
"fn*" (let [params (in form 1)
|
||||
body-bindings (do
|
||||
(var bb @{})
|
||||
(loop [[k v] :pairs bindings] (put bb k v))
|
||||
(each p params
|
||||
(put bb (if (struct? p) (p :name) p) :jolt/local))
|
||||
bb)
|
||||
body-exprs (tuple/slice form 2)
|
||||
analyzed-body (map |(analyze-form $ body-bindings) body-exprs)
|
||||
n-body (length analyzed-body)]
|
||||
{:op :fn :params params
|
||||
:body (if (> n-body 1)
|
||||
{:op :do
|
||||
:statements (array/slice analyzed-body 0 (- n-body 1))
|
||||
:ret (last analyzed-body)}
|
||||
(first analyzed-body))})
|
||||
"let*" (let [bind-vec (in form 1)
|
||||
body-exprs (tuple/slice form 2)
|
||||
binding-pairs (do
|
||||
(var pairs @[])
|
||||
(var i 0)
|
||||
(let [n (length bind-vec)]
|
||||
(while (< i n)
|
||||
(let [sym-s (in bind-vec i)
|
||||
name (if (struct? sym-s) (sym-s :name) sym-s)
|
||||
val-form (if (< (+ i 1) n) (in bind-vec (+ i 1)) nil)
|
||||
val-ast (if val-form (analyze-form val-form bindings) {:op :const :val nil})]
|
||||
(array/push pairs {:name name :init val-ast})
|
||||
(+= i 2))))
|
||||
pairs)
|
||||
body-bindings (do
|
||||
(var bb @{})
|
||||
(loop [[k v] :pairs bindings] (put bb k v))
|
||||
(each bp binding-pairs
|
||||
(put bb (bp :name) :jolt/local))
|
||||
bb)
|
||||
analyzed-body (map |(analyze-form $ body-bindings) body-exprs)
|
||||
n-body (length analyzed-body)]
|
||||
{:op :let
|
||||
:binding-pairs binding-pairs
|
||||
:body (if (> n-body 1)
|
||||
{:op :do
|
||||
:statements (array/slice analyzed-body 0 (- n-body 1))
|
||||
:ret (last analyzed-body)}
|
||||
(first analyzed-body))})
|
||||
(let [f-ast (analyze-form first-form bindings)
|
||||
args (map |(analyze-form $ bindings) (tuple/slice form 1))]
|
||||
{:op :invoke :fn f-ast :args args}))
|
||||
(let [f-ast (analyze-form first-form bindings)
|
||||
args (map |(analyze-form $ bindings) (tuple/slice form 1))]
|
||||
{:op :invoke :fn f-ast :args args})))
|
||||
# Macro expansion: if ctx is provided and head resolves to a macro,
|
||||
# expand it and re-analyze the expanded form
|
||||
(if (and ctx head-name
|
||||
(not (special-form? head-name))
|
||||
(resolve-macro ctx first-form))
|
||||
(let [macro-var (resolve-macro ctx first-form)
|
||||
macro-fn (var-get macro-var)
|
||||
expanded (apply macro-fn (tuple/slice form 1))]
|
||||
(analyze-form expanded bindings ctx))
|
||||
(if head-name
|
||||
(match head-name
|
||||
"quote" {:op :quote :expr (in form 1)}
|
||||
"do" (let [all-statements (array/slice form 1)
|
||||
n (length all-statements)
|
||||
analyzed (map |(analyze-form $ bindings ctx) all-statements)]
|
||||
{:op :do
|
||||
:statements (array/slice analyzed 0 (- n 1))
|
||||
:ret (in analyzed (- n 1))})
|
||||
"if" {:op :if
|
||||
:test (analyze-form (in form 1) bindings ctx)
|
||||
:then (analyze-form (in form 2) bindings ctx)
|
||||
:else (if (> (length form) 3)
|
||||
(analyze-form (in form 3) bindings ctx)
|
||||
{:op :const :val nil})}
|
||||
"def" {:op :def
|
||||
:name (in form 1)
|
||||
:init (analyze-form (in form 2) bindings ctx)}
|
||||
"fn*" (let [params (in form 1)
|
||||
body-bindings (do
|
||||
(var bb @{})
|
||||
(loop [[k v] :pairs bindings] (put bb k v))
|
||||
(each p params
|
||||
(put bb (if (struct? p) (p :name) p) :jolt/local))
|
||||
bb)
|
||||
body-exprs (tuple/slice form 2)
|
||||
analyzed-body (map |(analyze-form $ body-bindings ctx) body-exprs)
|
||||
n-body (length analyzed-body)]
|
||||
{:op :fn :params params
|
||||
:body (if (> n-body 1)
|
||||
{:op :do
|
||||
:statements (array/slice analyzed-body 0 (- n-body 1))
|
||||
:ret (last analyzed-body)}
|
||||
(first analyzed-body))})
|
||||
"let*" (let [bind-vec (in form 1)
|
||||
body-exprs (tuple/slice form 2)
|
||||
binding-pairs (do
|
||||
(var pairs @[])
|
||||
(var i 0)
|
||||
(let [n (length bind-vec)]
|
||||
(while (< i n)
|
||||
(let [sym-s (in bind-vec i)
|
||||
name (if (struct? sym-s) (sym-s :name) sym-s)
|
||||
val-form (if (< (+ i 1) n) (in bind-vec (+ i 1)) nil)
|
||||
val-ast (if val-form (analyze-form val-form bindings ctx) {:op :const :val nil})]
|
||||
(array/push pairs {:name name :init val-ast})
|
||||
(+= i 2))))
|
||||
pairs)
|
||||
body-bindings (do
|
||||
(var bb @{})
|
||||
(loop [[k v] :pairs bindings] (put bb k v))
|
||||
(each bp binding-pairs
|
||||
(put bb (bp :name) :jolt/local))
|
||||
bb)
|
||||
analyzed-body (map |(analyze-form $ body-bindings ctx) body-exprs)
|
||||
n-body (length analyzed-body)]
|
||||
{:op :let
|
||||
:binding-pairs binding-pairs
|
||||
:body (if (> n-body 1)
|
||||
{:op :do
|
||||
:statements (array/slice analyzed-body 0 (- n-body 1))
|
||||
:ret (last analyzed-body)}
|
||||
(first analyzed-body))})
|
||||
(let [f-ast (analyze-form first-form bindings ctx)
|
||||
args (map |(analyze-form $ bindings ctx) (tuple/slice form 1))]
|
||||
{:op :invoke :fn f-ast :args args}))
|
||||
(let [f-ast (analyze-form first-form bindings ctx)
|
||||
args (map |(analyze-form $ bindings ctx) (tuple/slice form 1))]
|
||||
{:op :invoke :fn f-ast :args args}))))
|
||||
|
||||
(tuple? form)
|
||||
(let [items (map |(analyze-form $ bindings) form)]
|
||||
(let [items (map |(analyze-form $ bindings ctx) form)]
|
||||
{:op :vector :items items})
|
||||
|
||||
(struct? form)
|
||||
|
|
@ -211,8 +339,7 @@
|
|||
|
||||
(var emit-ast nil)
|
||||
|
||||
(defn- emit-const-str
|
||||
[val buf]
|
||||
(defn- emit-const-str [val buf]
|
||||
(cond
|
||||
(nil? val) (buffer/push buf "nil")
|
||||
(= true val) (buffer/push buf "true")
|
||||
|
|
@ -313,30 +440,110 @@
|
|||
:quote (emit-quote-str (ast :expr) buf)
|
||||
(buffer/push buf (string "/* unhandled op: " (ast :op) " */")))))
|
||||
|
||||
# ============================================================
|
||||
# Emitter — AST → Janet data structure (for direct eval)
|
||||
# ============================================================
|
||||
|
||||
(var emit-expr nil)
|
||||
|
||||
(defn- emit-const-expr [val] val)
|
||||
|
||||
(defn- emit-do-expr [statements ret]
|
||||
(def exprs @['do])
|
||||
(each s statements (array/push exprs (emit-expr s)))
|
||||
(when ret (array/push exprs (emit-expr ret)))
|
||||
(tuple/slice (tuple ;exprs)))
|
||||
|
||||
(defn- emit-if-expr [test then else]
|
||||
(def exprs @['if])
|
||||
(array/push exprs (emit-expr test))
|
||||
(array/push exprs (emit-expr then))
|
||||
(when else (array/push exprs (emit-expr else)))
|
||||
(tuple/slice (tuple ;exprs)))
|
||||
|
||||
(defn- emit-def-expr [name-sym init]
|
||||
['def (symbol (name-sym :name)) (emit-expr init)])
|
||||
|
||||
(defn- emit-fn-expr [params body]
|
||||
(def param-syms @[])
|
||||
(each p params
|
||||
(array/push param-syms (symbol (if (struct? p) (p :name) p))))
|
||||
['fn (tuple/slice (tuple ;param-syms)) (emit-expr body)])
|
||||
|
||||
(defn- emit-let-expr [binding-pairs body]
|
||||
(def bind-tuple @[])
|
||||
(each bp binding-pairs
|
||||
(array/push bind-tuple (symbol (bp :name)))
|
||||
(array/push bind-tuple (emit-expr (bp :init))))
|
||||
['let (tuple/slice (tuple ;bind-tuple)) (emit-expr body)])
|
||||
|
||||
(defn- emit-invoke-expr [f-ast args]
|
||||
(def exprs @[(emit-expr f-ast)])
|
||||
(each arg args (array/push exprs (emit-expr arg)))
|
||||
(tuple/slice (tuple ;exprs)))
|
||||
|
||||
(defn- emit-symbol-expr [name] (symbol name))
|
||||
(defn- emit-local-expr [name] (symbol name))
|
||||
|
||||
(defn- emit-core-symbol-expr [janet-name]
|
||||
(or (get core-fn-values janet-name)
|
||||
(error (string "Core fn not found: " janet-name))))
|
||||
|
||||
(defn- emit-qualified-symbol-expr [ns name]
|
||||
(error (string "Cannot eval qualified symbol at compile time: " ns "/" name)))
|
||||
|
||||
(defn- emit-vector-expr [items]
|
||||
(def exprs @[])
|
||||
(each item items (array/push exprs (emit-expr item)))
|
||||
(tuple/slice (tuple ;exprs)))
|
||||
|
||||
(defn- emit-map-expr [form] form)
|
||||
|
||||
(defn- emit-quote-expr [expr]
|
||||
['quote (analyze-form expr @{})])
|
||||
|
||||
(set emit-expr
|
||||
(fn [ast]
|
||||
(match (ast :op)
|
||||
:const (emit-const-expr (ast :val))
|
||||
:symbol (emit-symbol-expr (ast :name))
|
||||
:local (emit-local-expr (ast :name))
|
||||
:core-symbol (emit-core-symbol-expr (ast :janet-name))
|
||||
:qualified-symbol (emit-qualified-symbol-expr (ast :ns) (ast :name))
|
||||
:do (emit-do-expr (ast :statements) (ast :ret))
|
||||
:if (emit-if-expr (ast :test) (ast :then) (ast :else))
|
||||
:def (emit-def-expr (ast :name) (ast :init))
|
||||
:fn (emit-fn-expr (ast :params) (ast :body))
|
||||
:let (emit-let-expr (ast :binding-pairs) (ast :body))
|
||||
:invoke (emit-invoke-expr (ast :fn) (ast :args))
|
||||
:vector (emit-vector-expr (ast :items))
|
||||
:map (emit-map-expr (ast :form))
|
||||
:quote (emit-quote-expr (ast :expr))
|
||||
(error (string "Unhandled op: " (ast :op))))))
|
||||
|
||||
# ============================================================
|
||||
# Public API
|
||||
# ============================================================
|
||||
|
||||
(defn compile-form
|
||||
"Compile a Clojure form to a Janet source string."
|
||||
[form]
|
||||
(let [ast (analyze-form form @{})
|
||||
"Compile a Clojure form to a Janet source string.
|
||||
Pass ctx for macro expansion."
|
||||
[form &opt ctx]
|
||||
(default ctx nil)
|
||||
(let [ast (analyze-form form @{} ctx)
|
||||
buf @""]
|
||||
(emit-ast ast buf)
|
||||
(string buf)))
|
||||
|
||||
(defn eval-janet-source
|
||||
"Parse and evaluate a Janet source string.
|
||||
Uses the proper parser→produce→eval pipeline so special forms work."
|
||||
[source]
|
||||
(def p (parser/new))
|
||||
(parser/consume p source)
|
||||
(parser/eof p)
|
||||
(def form (parser/produce p))
|
||||
(eval form))
|
||||
(defn compile-ast
|
||||
"Compile a Clojure form to an eval-able Janet data structure.
|
||||
Core function symbols are resolved to actual function values."
|
||||
[form &opt ctx]
|
||||
(default ctx nil)
|
||||
(emit-expr (analyze-form form @{} ctx)))
|
||||
|
||||
(defn compile-and-eval
|
||||
"Compile a Clojure form to Janet source and evaluate it.
|
||||
Returns the result value."
|
||||
[form]
|
||||
(eval-janet-source (compile-form form)))
|
||||
"Compile a Clojure form and evaluate it as Janet.
|
||||
Emits Janet data structures with resolved core functions."
|
||||
[form ctx]
|
||||
(eval (compile-ast form ctx)))
|
||||
|
|
|
|||
|
|
@ -50,9 +50,8 @@
|
|||
(put cache ns-name cached))
|
||||
|
||||
(each form forms
|
||||
(let [janet-src (compile-form form)]
|
||||
(array/push cached janet-src)
|
||||
(eval-janet-source janet-src)))
|
||||
(array/push cached form)
|
||||
(compile-and-eval form ctx))
|
||||
ns-name)
|
||||
# Interpreter path
|
||||
(do
|
||||
|
|
|
|||
|
|
@ -102,7 +102,7 @@
|
|||
|
||||
(defn compile-eval-str [s]
|
||||
(let [form (parse-string s)]
|
||||
(compile-and-eval form)))
|
||||
(compile-and-eval form nil)))
|
||||
|
||||
(assert (= 42 (compile-eval-str "42")) "eval literal")
|
||||
(assert (= 2 (compile-eval-str "(inc 1)")) "eval inc")
|
||||
|
|
@ -141,3 +141,32 @@
|
|||
(print " passed")
|
||||
|
||||
(print "\nAll compiler Phase 3 tests passed!")
|
||||
|
||||
# ============================================================
|
||||
# 11. Macro expansion (Phase 4)
|
||||
# ============================================================
|
||||
(print "11: macro expansion...")
|
||||
(use ../src/jolt/api)
|
||||
|
||||
(let [ctx (init {:compile? true})]
|
||||
# defn expands via compiler, produces Janet def
|
||||
(eval-string ctx "(defn square [n] (* n n))")
|
||||
(assert (= 25 (eval-string ctx "(square 5)")) "defn via compiler")
|
||||
|
||||
# when macro
|
||||
(assert (= 42 (eval-string ctx "(when true 42)")) "when true")
|
||||
(assert (= nil (eval-string ctx "(when false 42)")) "when false")
|
||||
|
||||
# let macro
|
||||
(assert (= 30 (eval-string ctx "(let [x 10 y 20] (+ x y))")) "let macro")
|
||||
|
||||
# fn macro
|
||||
(assert (= 49 (eval-string ctx "((fn [x] (* x x)) 7)")) "fn macro")
|
||||
|
||||
# and/or
|
||||
(assert (= 3 (eval-string ctx "(and 1 2 3)")) "and")
|
||||
(assert (= 99 (eval-string ctx "(or nil false 99)")) "or"))
|
||||
|
||||
(print " passed")
|
||||
|
||||
(print "\nAll compiler Phase 4 tests passed!")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue