diff --git a/.dirge/memory/MEMORY.md b/.dirge/memory/MEMORY.md index 19fa6e7..e059137 100644 --- a/.dirge/memory/MEMORY.md +++ b/.dirge/memory/MEMORY.md @@ -1,5 +1,7 @@ -Compiler in `src/jolt/compiler.janet` (~470 lines). Three emit modes: 1) compile-form → Janet source string (debug/display) 2) compile-ast → Janet data structures with resolved fn values (for eval) 3) compile-and-eval → compile-ast + eval. core-renames maps Clojure→Janet STRING names. core-fn-values maps Janet string names→actual function VALUES (used by compile-ast). analyze-form [form bindings ctx] — ctx optional for macro expansion. Macro expansion: when head symbol resolves to macro var, apply macro fn to args and re-analyze expanded form. Ops: const, do, if, def, fn*, let*, invoke, symbol (local/core/qualified), quote, vector, map. Symbol classification: locals → core-symbol → symbol (shadowing works). compile-form and compile-ast accept optional ctx arg. -§ -Compiler tests in `test/compiler-test.janet` (172 lines, 11 test groups). Groups: 1-literals, 2-do, 3-if, 4-def, 5-fn, 6-let, 7-invoke, 8-local classification, 9-compile-and-eval round-trip, 10-compile flag, 11-macro expansion (defn, when, let, fn, and/or via compile?). All 317 tests pass. -§ Janet's `eval` runs in Janet's default environment and does NOT have access to symbols imported via `(use ...)` in the calling file. `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. FIX: emit Janet data structures where function VALUES are embedded directly (e.g. `[core-inc 1]`) rather than source strings `"(core-inc 1)"`. The `core-fn-values` table resolves Janet symbol names to actual function values at compile time. +§ +Jolt Compiler Architecture (Phases 1-6, dfa9874→c366963): Two-phase — analyze-form (Clojure form → annotated AST) → emit-ast (→ Janet source string) or emit-expr (→ Janet data structures for eval). analyze-form takes [form bindings &opt ctx]; ctx needed for macro expansion. Symbol classification: bindings first (:local), then core-renames (:core-symbol), then plain (:symbol). Two emitter paths: string (compile-form) and data structures (compile-ast). Core fn values resolved via core-fn-values table. compile-and-eval takes [form ctx]; pass nil for no macro ctx. Naming: Clojure - → core-sub. Key bugs found: emit-vector-expr must wrap with (tuple ...) — bare tuples eval as fn calls. make-symbol must treat / at pos 0 as unqualified. raw-form->janet converter for quote (don't re-analyze). core-renames entry for fn? was missing. +§ +eval-string dispatch (api.janet): When :compile? is true, stateful forms (defmacro, ns, deftype, defmulti, defmethod, require, in-ns) always use interpreter. All others (def, defn via macros, pure expressions) go through compile-and-eval. defn bug was caused by macros not being expanded — fixed in Phase 4 by routing macros through compiler's analyze-form which expands them at analyze time. +§ +Phase 6: 47 comprehensive compile-mode tests in test/phase6-final.janet. Collections, math, predicates, comparison, seq ops (map/filter/reduce/take/drop), special forms (let/if/loop/try/quote), macros (defn/when/and/or/fn/if-let), complex nesting. All 317 tests pass. Remaining: syntax-quote, set! compiler support. deftype, defmulti/defmethod routed to interpreter (stateful). diff --git a/.dirge/memory/PITFALLS.md b/.dirge/memory/PITFALLS.md index b0de493..95f166b 100644 --- a/.dirge/memory/PITFALLS.md +++ b/.dirge/memory/PITFALLS.md @@ -1,5 +1,5 @@ -Janet eval scope & source-to-source failure: `(eval '(core-inc 1))` fails with "unknown symbol core-inc" even when the file does `(use ./core)`. Janet's `eval` runs in the default environment and doesn't see `use`-imported symbols. NEVER emit Janet source strings for eval. Instead emit Janet data structures where function VALUES are embedded directly: `[core-inc 1]` not `"(core-inc 1)"`. Use a `core-fn-values` table to resolve names at compile time. The `(parse s)` function returns `[symbol, error-position]` — must use parser pipeline for proper eval: `parser/new→consume→eof→produce→eval`. Without `parser/eof`, numbers/literals return nil from produce. Janet's `eval` also doesn't support special forms natively in data structures — `(eval [if true 1 2])` fails with "unknown symbol if". Must parse strings, not eval tuples. +Janet eval data-structure constraints: (1) Bare tuples eval as fn calls — use `['tuple val1 val2]` not `[val1 val2]` for vector values. (2) Janet try: `[(tuple ;[err-sym]) handler-body]` NOT `(catch [err] body)` — catch is NOT a first-position keyword. (3) quote in compile-ast: use raw-form→janet converter (map Jolt reader structs → Janet symbols) — don't re-analyze. (4) eval runs in default env, doesn't see `use`-imported symbols. FIX: embed fn values directly via core-fn-values table. § When you need to mutate a local with `set`, use `(var x nil)` not `(def x nil)`. `def` creates constants — `(set ns-name ...)` on a `def` fails with "cannot set constant". This hit us in loader.janet ns-name extraction loop. § -core-renames MUST match actual function names defined in core.janet. `"-"` maps to `core-sub` NOT `core--`. `"not"` maps to `core-not` (defined as `(defn core-not ...)`). Missing entries cause silent nil returns. When adding to core-renames, grep core.janet for the actual `(defn core-XXX ...)` or `(def core-XXX ...)` name. Then add matching entry to core-fn-values. +core-renames MUST match actual function names in core.janet. `"-"`→`core-sub` NOT `core--`. `"fn?"` was missing entirely (caused silent nil). Missing entries → symbol treated as unknown global, returns nil. When adding: grep core.janet for actual `(defn core-XXX)` name, add to BOTH core-renames (string table) and core-fn-values (fn value table). diff --git a/.dirge/skills/.usage.json b/.dirge/skills/.usage.json index 9e8da8f..5a52239 100644 --- a/.dirge/skills/.usage.json +++ b/.dirge/skills/.usage.json @@ -1,26 +1,4 @@ { - "jolt-dev": { - "created_by": "agent", - "use_count": 26, - "view_count": 37, - "patch_count": 32, - "last_used_at": "2026-06-02T19:38:51.833410+00:00", - "last_viewed_at": "2026-06-02T19:38:51.825602+00:00", - "last_patched_at": "2026-06-02T19:39:44.919664+00:00", - "created_at": "2026-06-01T21:26:06.614465+00:00", - "state": "active", - "pinned": false - }, - "jolt-compiler": { - "created_by": "agent", - "use_count": 0, - "view_count": 0, - "patch_count": 1, - "last_patched_at": "2026-06-02T19:40:03.526270+00:00", - "created_at": "2026-06-02T17:54:38.690279+00:00", - "state": "active", - "pinned": false - }, "jolt-bootstrap": { "created_by": "agent", "use_count": 9, @@ -42,5 +20,27 @@ "created_at": "2026-06-01T20:56:39.144222+00:00", "state": "active", "pinned": false + }, + "jolt-compiler": { + "created_by": "agent", + "use_count": 0, + "view_count": 0, + "patch_count": 1, + "last_patched_at": "2026-06-02T19:40:03.526270+00:00", + "created_at": "2026-06-02T17:54:38.690279+00:00", + "state": "active", + "pinned": false + }, + "jolt-dev": { + "created_by": "agent", + "use_count": 28, + "view_count": 39, + "patch_count": 33, + "last_used_at": "2026-06-02T20:12:45.918073+00:00", + "last_viewed_at": "2026-06-02T20:12:45.904425+00:00", + "last_patched_at": "2026-06-02T20:13:13.431432+00:00", + "created_at": "2026-06-01T21:26:06.614465+00:00", + "state": "active", + "pinned": false } } \ No newline at end of file diff --git a/.dirge/skills/jolt-dev/SKILL.md b/.dirge/skills/jolt-dev/SKILL.md index 8eb08c6..e040ce7 100644 --- a/.dirge/skills/jolt-dev/SKILL.md +++ b/.dirge/skills/jolt-dev/SKILL.md @@ -59,6 +59,28 @@ The match arm receives `ctx`, `bindings`, and `form` (the full list). Use `(in f ### Current special forms (22): `quote`, `syntax-quote`, `unquote`, `unquote-splicing`, `do`, `if`, `def`, `defmacro`, `fn*`, `let*`, `loop*`, `recur`, `throw`, `try`, `set!`, `var`, `locking`, `instance?`, `defmulti`, `defmethod`, `deftype`, `new`, `.` +## Compiler Development + +### Architecture +`src/jolt/compiler.janet` (721 lines). Two emitter paths: +- `compile-form` → `emit-ast` → Janet source string (debug/display) +- `compile-and-eval` → `emit-expr` → Janet data structures (direct eval, resolved fn values) + +### Adding a compiled op +1. **analyze-form**: add `match head-name` arm returning `{:op :your-op ...}` +2. **emit-ast**: add str function + `:your-op` case in `set emit-ast` dispatch +3. **emit-expr**: add expr function + `:your-op` case in `set emit-expr` dispatch +4. Add tests in `test/compiler-test.janet` + +### Emit-expr critical rules +- **Vectors**: wrap with `['tuple ...]` — bare tuples eval as fn calls +- **try/catch**: `[(tuple ;[err-sym]) handler]` NOT `(catch [err] body)` +- **quote**: use `raw-form->janet` converter, don't re-analyze +- **Core fns**: resolve via `core-fn-values` table, embed fn VALUES not names + +### Macro expansion +`analyze-form` checks `resolve-macro` first — if head is a macro var, applies fn, re-analyzes expanded form (only when ctx passed). + ## Persistent Data Structures Located in: diff --git a/src/jolt/compiler.janet b/src/jolt/compiler.janet index 1ecc9b8..a931b98 100644 --- a/src/jolt/compiler.janet +++ b/src/jolt/compiler.janet @@ -70,6 +70,9 @@ "take-while" "core-take-while" "drop-while" "core-drop-while" "nth" "core-nth" + "list" "core-list" + "name" "core-name" + "subs" "core-subs" "reverse" "core-reverse" "into" "core-into" "merge" "core-merge" @@ -216,7 +219,10 @@ (put t "core-apply" apply) (put t "core-some" core-some?) (put t "core-pr-str" core-str) - (put t "core-nth" core-get) + (put t "core-nth" core-nth) + (put t "core-list" core-list) + (put t "core-name" core-name) + (put t "core-subs" core-subs) t)) # Loop counter for generating unique loop function names diff --git a/src/jolt/core.janet b/src/jolt/core.janet index 6787c31..55dc251 100644 --- a/src/jolt/core.janet +++ b/src/jolt/core.janet @@ -352,6 +352,15 @@ (-- i)) (if (tuple? coll) (tuple/slice (tuple ;result)) result)) +(defn core-nth + "Return the nth element of a sequential collection." + [coll idx &opt default] + (if (and (>= idx 0) (< idx (length coll))) + (in coll idx) + (if (nil? default) + (error (string "Index " idx " out of bounds, length: " (length coll))) + default))) + (defn core-sort [coll] (let [arr (if (tuple? coll) (array/slice coll) coll) sorted (sort arr)] @@ -1017,6 +1026,7 @@ "drop-while" core-drop-while "concat" core-concat "reverse" core-reverse + "nth" core-nth "sort" core-sort "sort-by" core-sort-by "distinct" core-distinct diff --git a/test/phase6-final.janet b/test/phase6-final.janet index a5a5ab3..f6463ed 100644 --- a/test/phase6-final.janet +++ b/test/phase6-final.janet @@ -6,6 +6,7 @@ (let [ctx (init {:compile? true})] (print " collections...") + (assert (= :a (ct-eval ctx "(nth [:a :b :c :d] 0)")) "nth") (assert (= true (ct-eval ctx "(vector? [1 2])")) "vector?") (assert (= true (ct-eval ctx "(map? {:a 1})")) "map?") (assert (= true (ct-eval ctx "(fn? inc)")) "fn?")