From 09b4e3e1bd23e60ad7a17c8073618ecbae7f5ca4 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Wed, 3 Jun 2026 12:30:11 -0400 Subject: [PATCH] =?UTF-8?q?Phase=2011:=20Fix=20pre-existing=20failures=20?= =?UTF-8?q?=E2=80=94=20316/317=20passing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - types.janet: ns? now accepts both structs and tables (defensive for namespace-like tables created via @{...}) - core.janet: wire comment macro into core-bindings (was in core-macro-names but missing from core-bindings) - sci/lang_stubs.clj: minimal SCI type stubs for bootstrap (IBox, HasName, IVar, DynVar protocols, SciUnbound/Var/Namespace deftypes) - test-load-sci.janet: load stubs before SCI source files, removed broken preprocessor Before: 315 ok, 2 fail (SciVar + array/buffer) After: 316 ok, 1 fail (only deftype in lang.cljc remains) The remaining failure is lang.cljc's SciVar deftype with #?@ inserts for JVM/CLJS-specific protocol implementations — a known limitation for Phase 15 (SCI bootstrap). --- .dirge/memory/MEMORY.md | 10 ++-- .dirge/memory/PITFALLS.md | 6 +- .dirge/skills/.usage.json | 90 ++++++++++++++-------------- .dirge/skills/jolt-compiler/SKILL.md | 42 ++++++++++++- .dirge/skills/jolt-dev/SKILL.md | 34 ++++++++++- .dirge/skills/jolt-gotchas/SKILL.md | 17 +++--- src/jolt/clojure/sci/lang_stubs.clj | 32 ++++++++++ src/jolt/core.janet | 1 + src/jolt/types.janet | 2 +- test/test-load-sci.janet | 14 +++++ 10 files changed, 179 insertions(+), 69 deletions(-) create mode 100644 src/jolt/clojure/sci/lang_stubs.clj diff --git a/.dirge/memory/MEMORY.md b/.dirge/memory/MEMORY.md index bedec3e..a3c769a 100644 --- a/.dirge/memory/MEMORY.md +++ b/.dirge/memory/MEMORY.md @@ -1,9 +1,7 @@ -REPL: `main.janet` initializes context and sets current ns to "user". `print-value` renders scalars inline (prin) and collections via `print-collection` which recursively calls print-value for nested rendering. Collections: tuples→[v1 v2], arrays→(v1 v2), structs→{k v}, deftype tables→{k v} (filtering :jolt/deftype :cnt :buckets :_meta :jolt/type :phm), sets→#{v}. Jolt symbol structs render as `name` or `ns/name`. -§ -REPL collection rendering: print-value in main.janet uses cond with print-value→print-collection mutual recursion (needs forward var declaration). Tuples→[v1 v2], arrays→(v1 v2), structs→{k v}, sets→#{v}, keywords→:kw, symbols→name or ns/name. Use prin for scalars, print only after collection closing bracket. REPL starts in wrong ns after loading persistent structures — must ctx-set-current-ns to "user" after init. -§ -fn* special form dispatch: fn* form emitted by macros MUST be @[...] (array) to enter eval-list's special form match. If wrapped in [...] (tuple), eval-form hits the (tuple? form) branch which maps over items instead of dispatching fn*. Same applies to register-method, protocol-dispatch, and other special form calls emitted by macros — all must be array-wrapped. -§ Janet's struct? returns true for tuples — cond forms in print-value/eval-form MUST check (tuple? x) before (struct? x) or (get x :key). Otherwise Janet sees a tuple, says yes to struct?, and calls (get tuple :name) which fails with "expected integer key for tuple in range [0, N), got :name". This hit us in print-value rendering and eval-form struct handling. § Protocol system: Type registry in context env (:type-registry) maps type-tag→proto-name→method-name→fn. Three dispatch special forms: protocol-dispatch (resolves method via registry or reified methods), register-method (stores impl in registry), make-reified (creates anonymous object with :jolt/protocol-methods). fn* forms emitted by extend-type/extend-protocol MUST be @[...] (array) for eval-list dispatch. Protocols are maps with :jolt/type :jolt/protocol and :methods map. +§ +Project stats after Phase 10: 7,370 total lines across 35 source/test files. Key sources: compiler.janet (869 lines), evaluator.janet (869 lines), core.janet (1373 lines), types.janet (441 lines), reader.janet (463 lines), main.janet (125 lines), api.janet (93 lines), loader.janet (79 lines), phm.janet (199 lines). Test suite: 315 ok, 2 fail (pre-existing SciVar bootstrap + array/table/buffer errors). 9 .clj standard library modules under src/jolt/clojure/ and src/jolt/jolt/. All tests run via `jpm test`. +§ +REPL print-value uses buffer-based output: write-value/v buf appends formatted strings via buffer/push-string, then print-value creates buffer, builds string, and does a single (print (string buf)). This prevents Janet C runtime from interleaving native output between prin statements in jpm build executables. Cond catch-all must use true clause: Janet's cond treats plain expressions as tests, so (push-str buf (string v)) at end of cond would be evaluated as a test — need true before it. diff --git a/.dirge/memory/PITFALLS.md b/.dirge/memory/PITFALLS.md index b586d83..c1a0f73 100644 --- a/.dirge/memory/PITFALLS.md +++ b/.dirge/memory/PITFALLS.md @@ -1,5 +1,5 @@ -Janet's `case` for multi-arity dispatch: `(defn f [& args] (case (length args) 1 ... 2 ...))`. Used in core-derive, core-isa?, core-ancestors, core-descendants because Janet doesn't support Clojure-style `([arg1] body1) ([arg1 arg2] body2)` multi-arity defn syntax. +Janet's `break` does NOT return a value from a loop — `(break val)` in a `while` loop just exits, discarding val. To return from a loop, set a variable before break: `(do (set result val) (break))`. Hit us in phm-bucket-assoc where `(break nb)` silently returned nil instead of the new bucket. Fixed by capturing index before break and building the result array after the loop. § -Janet's boolean function doesn't exist — use (if x true false). Janet's defn doesn't support Clojure-style multi-arity syntax ([args] body) — use [& args] with case (length args) dispatch. fn? exists as Janet builtin (not Jolt core fn) — use (or (function? x) (cfunction? x)) in tests. +Clojure .clj source files loaded via eval-form cannot have docstrings. If a defn form has 5 elements (defn, name, docstring, params, body), the evaluator's defn macro handler gets 4 args instead of 3, breaking with "macro arity mismatch". All .clj files in src/jolt/clojure/ must use 4-element defn forms: (defn name [params] body). Docstrings on defn are a Clojure feature not supported by Jolt's defn macro. § -Janet's `cond` treats the last position as a test clause, NOT a catch-all body. A bare expression like `(push-str buf val)` in the last position runs as a test (always truthy, but executed for side effects between other cond clauses). Use `true (push-str buf val)` to make it a proper catch-all body. Hit us in buffer-based write-value — raw tuple addresses leaked into output because `(push-str buf (string v))` ran as a test clause between other branches. +PHM internal key leak: Core functions iterating over PHM maps with keys/pairs get internal metadata keys (:jolt/deftype, :cnt, :buckets, :_meta). Always check for set?/phm? first and use type-aware helpers (phm-to-struct, phs-seq, phm-keys, phm-entries) before generic iteration. Hit us in core-merge, core-reduce, core-every?, core-filter which all needed set?/phm? checks added. diff --git a/.dirge/skills/.usage.json b/.dirge/skills/.usage.json index 8f501ad..a180541 100644 --- a/.dirge/skills/.usage.json +++ b/.dirge/skills/.usage.json @@ -1,70 +1,70 @@ { - "jolt-compiler": { - "created_by": "agent", - "use_count": 10, - "view_count": 146, - "patch_count": 9, - "last_used_at": "2026-06-03T13:32:50.977662+00:00", - "last_viewed_at": "2026-06-03T14:17:26.562948+00:00", - "last_patched_at": "2026-06-03T03:03:27.524946+00:00", - "created_at": "2026-06-02T17:54:38.690279+00:00", - "state": "active", - "pinned": false - }, "jolt-bootstrap": { "created_by": "agent", "use_count": 9, - "view_count": 156, + "view_count": 243, "patch_count": 10, "last_used_at": "2026-06-02T18:45:23.336653+00:00", - "last_viewed_at": "2026-06-03T14:17:26.556567+00:00", + "last_viewed_at": "2026-06-03T16:29:39.837221+00:00", "last_patched_at": "2026-06-02T03:44:45.935676+00:00", "created_at": "2026-06-01T21:49:51.101718+00:00", "state": "active", "pinned": false }, + "jolt-gotchas": { + "created_by": "agent", + "use_count": 4, + "view_count": 222, + "patch_count": 4, + "last_used_at": "2026-06-03T16:12:06.850822+00:00", + "last_viewed_at": "2026-06-03T16:29:39.856662+00:00", + "last_patched_at": "2026-06-03T16:12:23.546648+00:00", + "created_at": "2026-06-03T03:50:41.474730+00:00", + "state": "active", + "pinned": false + }, + "jolt-dev": { + "created_by": "agent", + "use_count": 38, + "view_count": 272, + "patch_count": 46, + "last_used_at": "2026-06-03T16:12:27.833941+00:00", + "last_viewed_at": "2026-06-03T16:29:39.851827+00:00", + "last_patched_at": "2026-06-03T16:12:38.766457+00:00", + "created_at": "2026-06-01T21:26:06.614465+00:00", + "state": "active", + "pinned": false + }, "jpm-build": { "created_by": "agent", "use_count": 0, - "view_count": 147, + "view_count": 234, "patch_count": 0, - "last_viewed_at": "2026-06-03T14:17:26.587368+00:00", + "last_viewed_at": "2026-06-03T16:29:39.867842+00:00", "created_at": "2026-06-01T20:56:39.144222+00:00", "state": "active", "pinned": false }, + "jolt-compiler": { + "created_by": "agent", + "use_count": 11, + "view_count": 234, + "patch_count": 10, + "last_used_at": "2026-06-03T16:12:44.265835+00:00", + "last_viewed_at": "2026-06-03T16:29:39.843393+00:00", + "last_patched_at": "2026-06-03T16:13:25.725903+00:00", + "created_at": "2026-06-02T17:54:38.690279+00:00", + "state": "active", + "pinned": false + }, "jolt-persistent-structures": { - "created_by": "agent", - "use_count": 1, - "view_count": 133, - "patch_count": 0, - "last_used_at": "2026-06-03T03:49:43.520839+00:00", - "last_viewed_at": "2026-06-03T14:17:26.582273+00:00", - "created_at": "2026-06-03T03:35:04.130959+00:00", - "state": "active", - "pinned": false - }, - "jolt-dev": { - "created_by": "agent", - "use_count": 37, - "view_count": 184, - "patch_count": 44, - "last_used_at": "2026-06-03T13:32:50.995757+00:00", - "last_viewed_at": "2026-06-03T14:17:26.569463+00:00", - "last_patched_at": "2026-06-03T13:34:25.654327+00:00", - "created_at": "2026-06-01T21:26:06.614465+00:00", - "state": "active", - "pinned": false - }, - "jolt-gotchas": { "created_by": "agent", "use_count": 3, - "view_count": 134, - "patch_count": 3, - "last_used_at": "2026-06-03T13:35:01.902830+00:00", - "last_viewed_at": "2026-06-03T14:17:26.576167+00:00", - "last_patched_at": "2026-06-03T13:35:14.859032+00:00", - "created_at": "2026-06-03T03:50:41.474730+00:00", + "view_count": 222, + "patch_count": 0, + "last_used_at": "2026-06-03T16:13:31.440242+00:00", + "last_viewed_at": "2026-06-03T16:29:39.861788+00:00", + "created_at": "2026-06-03T03:35:04.130959+00:00", "state": "active", "pinned": false } diff --git a/.dirge/skills/jolt-compiler/SKILL.md b/.dirge/skills/jolt-compiler/SKILL.md index 4463384..fed1136 100644 --- a/.dirge/skills/jolt-compiler/SKILL.md +++ b/.dirge/skills/jolt-compiler/SKILL.md @@ -44,7 +44,7 @@ Order: qualified ns → local binding → core-symbol → bare symbol core-renames maps Clojure name strings → Janet function name strings (used by both emitter paths). core-fn-values maps Janet function name strings → actual function values (used by data-structure emitter only). -MUST keep both in sync. When adding a new core fn, update BOTH. +MUST keep both in sync. When adding a new core fn, update BOTH tables. **Missing entries → symbol treated as unknown global, returns nil.** Key name mappings: @@ -68,9 +68,9 @@ Key name mappings: ## Throw/try compilation - `throw` → `(error val)` in Janet -- `try/catch` → `(try body ([err] handler-body))` — Janet uses `([sym] handler)` format +- `try/catch` → `(try body ([err] handler-body))` — Janet uses `([sym] handler)` format, NOT `(catch sym handler)` -## emi-vector-expr critical fix +## emit-vector-expr critical fix **Bare tuples in Janet eval are function calls**: `[1 2 3]` tries to call `1` as a function. Always emit `['tuple ...]` or `(tuple ...)`. @@ -106,12 +106,48 @@ raw-form->janet converts Jolt symbols to Janet symbols, recursively for arrays/t (compile-and-eval form ctx)) ; compile ``` +## Protocol dispatch macros + +`core-extend-type` and `core-extend-protocol` emit `register-method` call forms. The fn* form MUST be `@[...]` (array) for `eval-list` to recognize it as a special form. Same for the outer call form wrapping `register-method`: + +```janet +(defn core-extend-type [type-sym proto-sym & impls] + (each method-spec impls + (def fn-form @[{:name "fn*"} arg-vec ;body]) ; @[...] array + (array/push result @[ + {:name "register-method"} ; @[...] array + type-sym proto-sym method-name fn-form]))) +``` + +## defprotocol method dispatch + +Protocol methods emit fn forms that delegate to `protocol-dispatch` special form. The fn form uses `@[...]` for fn* and its body so eval-list dispatches correctly: + +```janet +(def fn-form @[ + {:name "fn*"} + @[{:name "this"} {:name "&"} {:name "rest-args"}] + @[{:name "protocol-dispatch"} + {:name "quote"} protocol-name + {:name "quote"} method-name + {:name "this"} + {:name "rest-args"}]]) +``` + +In register-method, args (type-sym, proto-sym, method-name) are passed raw — evaluator resolves via `(in form N)`. + +## PHM integration in core functions + +~16 core functions have PHM-aware branches before generic struct/table handling: +`core-get`, `core-assoc`, `core-dissoc`, `core-conj`, `core-contains?`, `core-count`, `core-keys`, `core-vals`, `core-merge`, `core-merge-with`, `core-empty?`, `core-seq`, `core-into`, `core-map?`, `core-=`, `core-hash-map`. Sets later added to 6 of these. + ## Test files - `test/compiler-test.janet` — Phases 0-4 tests (source output + compile-eval + macro tests) - `test/phase6-final.janet` — Phase 6 comprehensive compile-mode tests (47 assertions) - `test/phase5-test.janet` — Phase 5 multimethod + hierarchy tests - `test/phase6-test.janet` — Phase 6 reader extension tests +- `test/phase8-test.janet` — Phase 8 protocol system tests - `test/hash-map-test.janet` — Phase 2 PersistentHashMap tests Run: `janet test/.janet` or `jpm test` \ No newline at end of file diff --git a/.dirge/skills/jolt-dev/SKILL.md b/.dirge/skills/jolt-dev/SKILL.md index cf49cad..e6514c8 100644 --- a/.dirge/skills/jolt-dev/SKILL.md +++ b/.dirge/skills/jolt-dev/SKILL.md @@ -18,7 +18,37 @@ jpm test # runs all tests janet test/foo.janet # run a single test file from project root ``` -## Special Form Checklist +## Testing Patterns + +```bash +# Single test file +janet test/compiler-test.janet + +# Full suite +jpm test + +# Phase-specific tests +janet test/phase5-test.janet # multimethods +janet test/phase8-test.janet # protocol system +janet test/phase10-test.janet # standard library + +# REPL test — pipe expressions in +printf "(range 10)\n[1 2 3]\n{:a 1}\n" | janet src/jolt/main.janet +``` + +## Loading .clj Files + +`.clj` files are loaded via `eval-form` in the interpreter: +```janet +(def src (slurp "src/jolt/clojure/string.clj")) +(var remaining src) +(while (> (length (string/trim remaining)) 0) + (def [form rest] (parse-next remaining)) + (set remaining rest) + (when form (eval-form ctx @{} form))) +``` + +**Critical constraint:** .clj files must NOT have docstrings on defn forms. Jolt's defn macro only handles 4-element forms: `(defn name [params] body)`. A 5-element form `(defn name "doc" [params] body)` causes "macro arity mismatch". To add a new special form to the evaluator: @@ -31,7 +61,7 @@ The match arm receives `ctx`, `bindings`, and `form` (the full list). Use `(in f **Non-symbol heads** (keywords, etc.): `eval-list` first checks `(and (struct? first-form) (= :symbol (...)))` before extracting `name`. If not a symbol, falls through to default function application. ### Current special forms (37): -`quote`, `syntax-quote`, `unquote`, `unquote-splicing`, `do`, `if`, `def`, `defmacro`, `fn*`, `let*`, `loop*`, `recur`, `throw`, `try`, `set!`, `var`, `locking`, `instance?`, `defmulti`, `defmethod`, `deftype`, `new`, `.`, `var-get`, `var-set`, `var?`, `alter-var-root`, `find-var`, `intern`, `alter-meta!`, `reset-meta!`, `disj`, `set?`, `protocol-dispatch`, `register-method`, `make-reified`, `satisfies?` +`quote`, `syntax-quote`, `unquote`, `unquote-splicing`, `do`, `if`, `def`, `defmacro`, `fn*`, `let*`, `loop*`, `recur`, `throw`, `try`, `set!`, `var`, `locking`, `instance?`, `defmulti`, `defmethod`, `deftype`, `new`, `.`, `var-get`, `var-set`, `var?`, `alter-var-root`, `find-var`, `intern`, `alter-meta!`, `reset-meta!`, `disj`, `set?`, `satisfies?`, `protocol-dispatch`, `register-method`, `make-reified` ## Compiler Architecture diff --git a/.dirge/skills/jolt-gotchas/SKILL.md b/.dirge/skills/jolt-gotchas/SKILL.md index 976b09a..2687753 100644 --- a/.dirge/skills/jolt-gotchas/SKILL.md +++ b/.dirge/skills/jolt-gotchas/SKILL.md @@ -54,15 +54,14 @@ A bare expression in the last position of `cond` is treated as a **test** clause Without `true`, the last expression executes as a side-effect test between branches. Hit us in buffer-based write-value — raw tuple addresses leaked into REPL output. -## Janet `cond` Requires `true` Guard for Catch-All +## REPL: Buffer-Based Output Prevents C-Runtime Interleaving -A bare expression in the last position of `cond` is treated as a **test** clause (not body). It executes between other branches as a side-effect test. Use `true` as the test to make it a proper catch-all: +Janet's C runtime in `jpm build` executables interleaves native `` output between `prin` statements. Solution: build entire output string in a buffer, then output atomically with a single `print` call. Use `write-value/v buf` + `print-value` creates buffer → `print (string buf)`. -```janet -(cond - (nil? x) (buf "nil") - (number? x) (buf (string x)) - true (buf (string x))) ; ← `true` required -``` +## Janet `struct?` Returns `true` for Tuples -Without `true`, `(push-str buf (string v))` in the last position leaked raw tuple addresses into REPL output. \ No newline at end of file +Always check `(tuple? x)` BEFORE `(struct? x)` in cond forms. Otherwise `(get tuple :key)` fails with "expected integer key for tuple in range [0, N), got :key". Hit us in `print-value` (symbol check on tuples) and `eval-form` struct handling. + +## PHM Internal Key Leakage + +PHM and set internal keys (`:jolt/deftype`, `:cnt`, `:buckets`, `:_meta`, `:jolt/type`, `:phm`) leak into `pairs`/`keys` iteration. Core fns that iterate collections (`core-merge`, `core-reduce`, `core-every?`, `core-filter`) must check for `set?`/`phm?` first and use type-aware helpers (`phm-to-struct`, `phs-seq`, `phm-keys`, `phm-entries`) before generic iteration. \ No newline at end of file diff --git a/src/jolt/clojure/sci/lang_stubs.clj b/src/jolt/clojure/sci/lang_stubs.clj new file mode 100644 index 0000000..b5b78b4 --- /dev/null +++ b/src/jolt/clojure/sci/lang_stubs.clj @@ -0,0 +1,32 @@ +; Minimal SCI type stubs for Jolt bootstrap +; Provides the deftype definitions that SCI's Clojure source references +; before the full SCI runtime is loaded. + +; Protocol stubs for SCI's type system +(defprotocol IBox (setVal [this v]) (getVal [this])) +(defprotocol HasName (getName [this])) +(defprotocol IVar (bindRoot [this v]) (getRawRoot [this]) (toSymbol [this]) + (isMacro [this]) (setThreadBound [this v]) (unbind [this]) (hasRoot [this])) +(defprotocol DynVar (dynamic? [this])) +(defprotocol IReified (getInterfaces [this]) (getMethods [this]) + (getProtocols [this]) (getFields [this])) + +; Unbound sentinel for vars +(deftype SciUnbound [the-var]) + +; Core SCI types — keep minimal for bootstrap +(deftype Namespace [name mappings aliases imports]) +(deftype Var [root name meta macro dynamic ns]) + +; Store stub (sci.ctx-store provides a global context atom) +(def ctx-store (atom nil)) + +; Macro helpers from sci.impl.macros +(defn deftime [& body] nil) +(defn usetime [& body] (eval (first body))) +(defmacro ? [& args] + (if (contains? &env (quote &env)) + (let [form (first args)] + (if (= :clj (first form)) + (second form) + (if (= :cljs (first form)) nil))))) diff --git a/src/jolt/core.janet b/src/jolt/core.janet index ab3a02d..5d7f2e0 100644 --- a/src/jolt/core.janet +++ b/src/jolt/core.janet @@ -1334,6 +1334,7 @@ "IllegalStateException" core-IllegalStateException "definterface" core-definterface "defrecord" core-defrecord + "comment" core-comment "resolve" core-resolve "update" core-update "copy-core-var" core-copy-core-var diff --git a/src/jolt/types.janet b/src/jolt/types.janet index f050c6e..40f4d2d 100644 --- a/src/jolt/types.janet +++ b/src/jolt/types.janet @@ -219,7 +219,7 @@ (defn ns? "Check if x is a Jolt Namespace." [x] - (and (struct? x) (= :jolt/namespace (x :jolt/type)))) + (and (or (struct? x) (table? x)) (= :jolt/namespace (x :jolt/type)))) (defn ns-name "Return the name symbol of a namespace." diff --git a/test/test-load-sci.janet b/test/test-load-sci.janet index 2337ce6..b07712d 100644 --- a/test/test-load-sci.janet +++ b/test/test-load-sci.janet @@ -5,6 +5,20 @@ (def ctx (init)) +(printf "Loading SCI stubs...\n") +(defn load-stubs [ctx filepath] + (var s (slurp filepath)) + (var count 0) + (while (> (length (string/trim s)) 0) + (def [form rest] (parse-next s)) + (set s rest) + (++ count) + (when (not (nil? form)) + (eval-form ctx @{} form))) + (printf " Loaded %d stub forms\n" count)) + +(load-stubs ctx "src/jolt/clojure/sci/lang_stubs.clj") + (defn load-file [ctx path] (var s (slurp path)) (var count 0)