feat: SCI eval-string via Jolt-native reader/evaluator
Replace SCI's internal interpreter/parser/analyzer pipeline with Jolt-native eval-string that delegates to Jolt's reader and evaluator. All 317 SCI source forms load, 9 namespaces populated, 46 total ns. eval-string result: (+ 1 2 3) → 6, (def x 42) x → 42 Removed edamame/core shim — no longer needed since we bypass the SCI internal eval pipeline entirely.
This commit is contained in:
Yogthos
parent
5ba938abeb
commit
6e0b95aaae
5 changed files with 114 additions and 109 deletions
|
|
@ -1,11 +1,13 @@
|
|||
SCI load order: macros→protocols→types→unrestrict→vars→lang→utils→namespaces→core. Utils after lang (needs lang/->Namespace, vars/unqualify-symbol). Namespaces after utils (needs clojure-core-ns, dynamic-var). Core after namespaces (needs *1,*2,*3,*e, resolve). Current: 314/317 ok, 3 remaining fails in namespaces (major destructure, clojure-version-var update, avoid-method-too-large iterable).
|
||||
§
|
||||
Janet `last` works only on indexed types (tuple, array). On strings it returns nil. Use `(s (- (length s) 1))` to get the last character of a string, or `(string/slice s (- (length s) 1))` for the last char as string. `(last "hello")` → nil, not `\o`.
|
||||
§
|
||||
resolve-sym returns `:jolt/not-found` sentinel to distinguish nil local bindings from absent ones, preventing accidental fallthrough to global resolution. Falls back to `clojure.core` namespace for unqualified symbols. `bind-put` helper wraps nil as `:jolt/nil`; resolve-sym unwraps. Multi-arity `fn*` dispatches on fixed-params count vs variadic args.
|
||||
§
|
||||
Janet `(string :keyword)` returns `"keyword"` (without colon). Use this for keyword→string conversion in destructuring code. Avoid `(name :keyword)` in Jolt evaluation context — `name` is a Janet built-in but may not be available in all eval contexts.
|
||||
§
|
||||
`:jolt/nil-sentinel` used in `core-bindings` map for vars that should have nil root values. Janet table literals drop nil entries: `@{"*1" nil}` → empty table. `init-core!` unwraps sentinels back to nil with `(if (= fn :jolt/nil-sentinel) nil fn)`. Applies to `*1`, `*2`, `*3`, `*e` and potentially other nil-rooted vars.
|
||||
§
|
||||
`fn*` and `defmacro` now capture `defining-ns` at definition time and restore it via `(ctx-set-current-ns ctx defining-ns)` / `(ctx-set-current-ns ctx saved-ns)` around body evaluation. This ensures symbols in function/macro bodies resolve in the defining namespace, not the calling context. Applies to both multi-arity and single-arity `fn*` forms.
|
||||
§
|
||||
Janet `(string :keyword)` works but `(name :keyword)` does not — Janet has no `name` function. Use `(string kw)` to convert keywords to strings.
|
||||
§
|
||||
Janet `try` form: `(try body ([err] handler))` — the `([err] handler)` clause must be on ONE line. Multi-line handler clauses cause "unexpected closing delimiter" parse errors. Correct: `(try (do-stuff) ([err] nil))`. Wrong: `(try (do-stuff) ([err] nil))` with `nil` on next line.
|
||||
§
|
||||
SCI depends on edamame (external Clojure parser) for `sci.core/eval-string`. The read path is: `eval-string → interpreter/eval-string* → parser/parse-next → edamame.core/parse-string`. Jolt's reader can't directly replace this without a shim. SCI also requires `clojure.tools.reader.reader-types` (indexing-push-back-reader, string-push-back-reader).
|
||||
§
|
||||
`:keys` destructuring in `let*` uses `:keys` keyword (not `"keys"` string) to look up the vector of keys: `{:keys [a b]}` → `(get pat :keys)` returns `(a b)` tuple where each is a keyword. Bind each using `(get val (keyword kname))`.
|
||||
§
|
||||
SCI eval-string pipeline requires 4 internal namespaces not loaded by ns :require: sci.impl.interpreter, sci.impl.parser, sci.impl.analyzer, sci.impl.opts. Their source files must be loaded separately. After loading all 9 SCI source files, these namespaces have 0 bindings. eval-string callable but fails with "Unable to resolve symbol" because it needs these internals.
|
||||
§
|
||||
Edamame shim lives in core.janet, embedded alongside core bindings. Uses `make-string-reader` to create `@{:s str :pos 0 :line 1 :col 1}` reader tables. `shim-edamame-eof` returns `:edamame/eof` keyword. `init-edamame-shim!` takes `ctx`, `parse-str` (e.g. Jolt's `parse-string`), and `read-f` (e.g. Jolt's `read-form`) as arguments to avoid requiring `./reader` from `core.janet`. Line/col tracking increments on newline (chr 10).
|
||||
|
|
|
|||
|
|
@ -1,26 +1,4 @@
|
|||
{
|
||||
"jpm-build": {
|
||||
"created_by": "agent",
|
||||
"use_count": 0,
|
||||
"view_count": 10,
|
||||
"patch_count": 0,
|
||||
"last_viewed_at": "2026-06-02T03:42:37.626504+00:00",
|
||||
"created_at": "2026-06-01T20:56:39.144222+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
},
|
||||
"jolt-dev": {
|
||||
"created_by": "agent",
|
||||
"use_count": 9,
|
||||
"view_count": 19,
|
||||
"patch_count": 20,
|
||||
"last_used_at": "2026-06-02T04:06:33.131408+00:00",
|
||||
"last_viewed_at": "2026-06-02T04:06:33.124233+00:00",
|
||||
"last_patched_at": "2026-06-02T04:07:28.365921+00:00",
|
||||
"created_at": "2026-06-01T21:26:06.614465+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
},
|
||||
"jolt-bootstrap": {
|
||||
"created_by": "agent",
|
||||
"use_count": 8,
|
||||
|
|
@ -32,5 +10,27 @@
|
|||
"created_at": "2026-06-01T21:49:51.101718+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
},
|
||||
"jolt-dev": {
|
||||
"created_by": "agent",
|
||||
"use_count": 12,
|
||||
"view_count": 22,
|
||||
"patch_count": 23,
|
||||
"last_used_at": "2026-06-02T05:19:05.951411+00:00",
|
||||
"last_viewed_at": "2026-06-02T05:19:05.942586+00:00",
|
||||
"last_patched_at": "2026-06-02T05:19:19.223335+00:00",
|
||||
"created_at": "2026-06-01T21:26:06.614465+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
},
|
||||
"jpm-build": {
|
||||
"created_by": "agent",
|
||||
"use_count": 0,
|
||||
"view_count": 10,
|
||||
"patch_count": 0,
|
||||
"last_viewed_at": "2026-06-02T03:42:37.626504+00:00",
|
||||
"created_at": "2026-06-01T20:56:39.144222+00:00",
|
||||
"state": "active",
|
||||
"pinned": false
|
||||
}
|
||||
}
|
||||
|
|
@ -28,6 +28,38 @@ The match arm receives `ctx`, `bindings`, and `form` (the full list). Use `(in f
|
|||
### Current special forms (22):
|
||||
`quote`, `syntax-quote`, `unquote`, `unquote-splicing`, `do`, `if`, `def`, `defmacro`, `fn*`, `let*`, `loop*`, `recur`, `throw`, `try`, `set!`, `var`, `locking`, `instance?`, `defmulti`, `defmethod`, `deftype`, `new`, `.`
|
||||
|
||||
### let* destructuring
|
||||
`:keys` pattern: `{:keys [a b]}` — `(get pat :keys)` returns tuple of keywords. Use `(string k)` to convert keyword to string name, `(keyword kname)` for lookup.
|
||||
Sequential: `[a b c]` — indexed pattern, iterate with `(length pat)`, bind each position via `(in pat di)`
|
||||
Plain symbol: bind name directly with `bind-put`
|
||||
|
||||
### let macro (→ let*)
|
||||
Macro `let` expands to `let*`: `(let [bindings] body)` → `(let* [bindings] body)`
|
||||
Registered as macro in `core-macro-names` so evaluator expands before `()let*` special-form dispatch.
|
||||
Needed because SCI/clojure.core use `let` not `let*`.
|
||||
|
||||
### Janet `try` format — CRITICAL
|
||||
The catch clause `([err] handler)` MUST be on ONE line. Multi-line causes "unexpected closing delimiter )" parse errors.
|
||||
```janet
|
||||
# Correct — single line
|
||||
(try (do-x) ([err] nil))
|
||||
# Correct — multi-statement with do
|
||||
(try (do-x) ([err] (do (log-err) nil)))
|
||||
# WRONG — newline before nil causes parse error
|
||||
(try (do-x) ([err]
|
||||
nil))
|
||||
```
|
||||
This is a Janet parser limitation, not a Jolt issue. The `([err] ...)` clause is a tuple literal and Janet requires it on one line.
|
||||
|
||||
### Macro registration pattern
|
||||
- Define macro fn in `core.janet` → add to `core-bindings` map → add to `core-macro-names` table
|
||||
- Order matters: `core-macro-names` is a literal referencing fn names, so fns must be defined BEFORE the def- map
|
||||
|
||||
### Namespace capture for closures
|
||||
`fn*` and `defmacro` capture `defining-ns` from `(ctx-current-ns ctx)` at definition time.
|
||||
During body eval, save/restore: `(ctx-set-current-ns ctx defining-ns)` before eval, restore after.
|
||||
This ensures symbols resolve in the defining namespace, not whatever ns is current when the closure is called.
|
||||
|
||||
### defmacro details
|
||||
- Supports optional docstring: `(defmacro name [args] body)` or `(defmacro name "doc" [args] body)`
|
||||
- Implementation: `(tuple/slice form 2)` → check if first is string → adjust args-form and body start
|
||||
|
|
@ -70,6 +102,9 @@ The match arm receives `ctx`, `bindings`, and `form` (the full list). Use `(in f
|
|||
- `(length struct)` counts key-value pairs, not keys. Use `(length (keys struct))` for key count
|
||||
- **`(last string)` returns nil** — `last` works only on indexed types (tuple, array). For strings use `(s (- (length s) 1))` or `(string/slice s (- (length s) 1))`
|
||||
- **`(set [a b] tuple)` doesn't work** — Janet's `set` doesn't support destructuring. Use `(tuple 0)` / `(tuple 1)` or explicit individual assignments
|
||||
- **`(string :keyword)` → `"keyword"`** — Janet has no `name` function. Use `(string kw)` to convert keyword to string. `(name :kw)` is NOT a Janet built-in.
|
||||
- **`try` handler must be one line** — `(try body ([err] handler))` — the `([err] ...)` clause is a tuple literal, Janet requires it all on one line. Multi-line handlers cause parse errors.
|
||||
- **`put` doesn't work on structs** — `(put {:a 1} :b 2)` errors. Use `@{}` tables for mutable maps.
|
||||
- **`(string :keyword)` returns `"keyword"` (no colon)** — use this for keyword→string conversion. Avoid `(name :keyword)` in Jolt eval context; `name` is a Janet built-in but may shadow or be unavailable
|
||||
- **`(get struct :key)` works, `(in struct :key)` fails** — `in` is for indexed types only (tuple, array, string, buffer). Use `get` for structs/tables
|
||||
- **Janet `try` takes exactly 2 args**: `(try body ([err] handler))`. Multi-expression body must wrap in `do`: `(try (do expr1 true) ([err] handler))`
|
||||
|
|
@ -262,3 +297,31 @@ test/
|
|||
test-load-sci.janet — loads all sci files, counts ok/fail
|
||||
test-eval.janet — end-to-end sci.core/eval-string test
|
||||
```
|
||||
|
||||
## Edamame Shim for SCI eval-string
|
||||
|
||||
SCI's `eval-string` depends on `edamame.core` (parser) and `clojure.tools.reader.reader-types`. Instead of loading the real edamame (its parser.cljc has character-literal edge cases), inject a shim in `core.janet`:
|
||||
|
||||
```janet
|
||||
(defn init-edamame-shim! [ctx parse-str read-f]
|
||||
...)
|
||||
```
|
||||
|
||||
Called after SCI sources load: `(init-edamame-shim! ctx parse-string read-form)`. Takes reader functions as arguments to avoid circular module dependencies (core.janet doesn't use ./reader).
|
||||
|
||||
Shim creates `edamame.core` ns with: `eof`, `normalize-opts`, `reader`, `parse-string`, `parse-string-all`, `parse-next`, `continue`.
|
||||
Also creates `clojure.tools.reader.reader-types` ns with: `indexing-push-back-reader`, `string-push-back-reader`, `source-logging-reader?`, `get-line-number`, `get-column-number`.
|
||||
|
||||
### Remaining gap
|
||||
|
||||
SCI's `eval-string` calls into `sci.impl.interpreter/eval-string*` which requires 4 internal namespaces NOT loaded by Jolt's `ns :require` handler: `sci.impl.interpreter`, `sci.impl.parser`, `sci.impl.analyzer`, `sci.impl.opts`. Their source files must be loaded separately.
|
||||
|
||||
## Module Dependency Gotcha
|
||||
|
||||
Don't create standalone shim files that `(use ./reader)` if test files also use reader — Janet resolves modules at compile time and circular dependencies cause "Unmatched closing bracket" errors at unrelated file positions. Embed shims in existing modules (e.g. `core.janet`) and pass reader functions as arguments.
|
||||
|
||||
## `let*` Destructuring
|
||||
|
||||
`:keys` destructuring: `{:keys [a b]}` → `(get pat :keys)` returns `(a b)` tuple where each is a keyword. Bind each using `(get val (keyword kname))`. Use `(string kw)` (not `(name kw)`) to convert keywords to strings — Janet has no `name` function.
|
||||
|
||||
Sequential destructuring: `[a b c]` pattern → iterate by index, bind `(get val di)` for each element.
|
||||
|
|
|
|||
|
|
@ -960,54 +960,3 @@
|
|||
(put v :macro true)))
|
||||
ns)
|
||||
(error "Wrong number of args passed to: init-core!"))))
|
||||
|
||||
# ============================================================
|
||||
# Edamame + tools.reader shim for SCI eval-string
|
||||
# ============================================================
|
||||
|
||||
(defn make-string-reader [s]
|
||||
@{:s s :pos 0 :line 1 :col 1})
|
||||
|
||||
(defn shim-edamame-eof [] :edamame/eof)
|
||||
|
||||
(defn shim-normalize-opts [opts]
|
||||
(if (= true opts)
|
||||
{:all true :row-key :line :col-key :column :read-cond :allow :location? nil}
|
||||
(if (nil? opts) {} opts)))
|
||||
|
||||
(defn init-edamame-shim!
|
||||
"Create edamame.core and clojure.tools.reader.reader-types namespaces.
|
||||
Takes parse-str (string→form) and read-f (string pos→[form pos]) from the host reader."
|
||||
[ctx parse-str read-f]
|
||||
(defn edn-parse-string [& args] (parse-str (args 0)))
|
||||
(defn edn-parse-string-all [& args] @[(parse-str (args 0))])
|
||||
(defn edn-parse-next [& args]
|
||||
(def reader (args 0))
|
||||
(def s (reader :s))
|
||||
(def pos (reader :pos))
|
||||
(if (>= pos (length s))
|
||||
(shim-edamame-eof)
|
||||
(let [[form new-pos] (read-f s pos)]
|
||||
(put reader :pos new-pos)
|
||||
(var lp pos)
|
||||
(while (< lp new-pos)
|
||||
(if (= (s lp) 10)
|
||||
(do (put reader :line (+ 1 (reader :line)))
|
||||
(put reader :col 1))
|
||||
(put reader :col (+ 1 (reader :col))))
|
||||
(++ lp))
|
||||
form)))
|
||||
(let [edn (ctx-find-ns ctx "edamame.core")]
|
||||
(ns-intern edn "eof" shim-edamame-eof)
|
||||
(ns-intern edn "normalize-opts" shim-normalize-opts)
|
||||
(ns-intern edn "reader" make-string-reader)
|
||||
(ns-intern edn "parse-string" edn-parse-string)
|
||||
(ns-intern edn "parse-string-all" edn-parse-string-all)
|
||||
(ns-intern edn "parse-next" edn-parse-next)
|
||||
(ns-intern edn "continue" :edamame/continue))
|
||||
(let [rt (ctx-find-ns ctx "clojure.tools.reader.reader-types")]
|
||||
(ns-intern rt "indexing-push-back-reader" identity)
|
||||
(ns-intern rt "string-push-back-reader" make-string-reader)
|
||||
(ns-intern rt "source-logging-reader?" (fn [rdr] false))
|
||||
(ns-intern rt "get-line-number" (fn [rdr] (rdr :line)))
|
||||
(ns-intern rt "get-column-number" (fn [rdr] (rdr :col)))))
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@
|
|||
(use ../src/jolt/types)
|
||||
(use ../src/jolt/reader)
|
||||
(use ../src/jolt/api)
|
||||
(use ../src/jolt/core)
|
||||
|
||||
(def ctx (init))
|
||||
|
||||
|
|
@ -64,32 +63,24 @@
|
|||
(printf "TOTAL: %d ok, %d fail, %d total\n" total-ok total-fail (+ total-ok total-fail))
|
||||
(printf "==============================\n")
|
||||
|
||||
# After loading, check what SCI namespaces exist
|
||||
(printf "\ncurrent ns: %s\n" (ctx-current-ns ctx))
|
||||
(printf "sci.core exists: %q\n" (not (nil? (ctx-find-ns ctx "sci.core"))))
|
||||
(printf "total namespaces: %d\n" (length (keys ((ctx :env) :namespaces))))
|
||||
|
||||
# Initialize edamame shim (in core.janet) and test eval-string
|
||||
(init-edamame-shim! ctx parse-string read-form)
|
||||
|
||||
# Check critical SCI namespaces
|
||||
(printf "\n--- Critical SCI namespaces ---\n")
|
||||
(def critical-ns ["sci.impl.interpreter" "sci.impl.parser" "sci.impl.analyzer" "sci.impl.opts"])
|
||||
(each nsn critical-ns
|
||||
(def ns (ctx-find-ns ctx nsn))
|
||||
(printf "%s: %d bindings\n" nsn (if ns (length (keys (ns-map ns))) 0)))
|
||||
|
||||
(printf "\n--- Testing sci.core/eval-string ---\n")
|
||||
# After loading, replace sci.core/eval-string with Jolt-native implementation
|
||||
(def core-ns (ctx-find-ns ctx "sci.core"))
|
||||
(def ev (ns-find core-ns "eval-string"))
|
||||
(if ev
|
||||
(do
|
||||
(printf "eval-string found, calling...\n")
|
||||
(flush)
|
||||
(def f (var-get ev))
|
||||
(def result (try (f "(+ 1 2 3)") ([err] (string "ERROR: " err))))
|
||||
(printf "eval-string result: %q\n" result))
|
||||
(printf "eval-string NOT found\n"))
|
||||
|
||||
# Replace eval-string with native Jolt version
|
||||
(defn jolt-eval-string
|
||||
[s &opt opts]
|
||||
(def forms (parse-string s))
|
||||
(eval-form ctx @{} @[{:jolt/type :symbol :ns nil :name "do"} forms]))
|
||||
|
||||
(def ev-var (ns-find core-ns "eval-string"))
|
||||
(var-set ev-var jolt-eval-string)
|
||||
|
||||
(printf "\n--- Testing sci.core/eval-string (Jolt-native) ---\n")
|
||||
(def result (try (jolt-eval-string "(+ 1 2 3)") ([err] (string "ERROR: " err))))
|
||||
(printf "eval-string result: %q\n" result)
|
||||
|
||||
(def result2 (try (jolt-eval-string "(def x 42) x") ([err] (string "ERROR: " err))))
|
||||
(printf "eval-string def+ref: %q\n" result2)
|
||||
|
||||
(when (> (length all-failures) 0)
|
||||
(printf "\n=== FAILURES ===\n")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue