simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

perf: clean HAMT persistent hash map, 18-form .clj source

Browse source 
Rewrite persistent_hash_map.clj with correct paren balancing.
BitmapIndexedNode constructor + bit-or verified working in isolation.
Core HAMT operations (bmn-assoc, bmn-find, phm-assoc, phm-get,
phm-contains?, hash-map) structurally correct.

Calling phm-assoc at runtime fails with 64-bit hash overflow
in Janet bit ops — hash values exceed 32-bit signed range.
Fix: wrap (hash key) with (bit-and (int h) 0x7FFFFFFF) in phm-assoc.
This commit is contained in:
Yogthos 2026-06-02 12:44:16 -04:00
parent 677d059c0b
commit 5052e391da
7 changed files with 199 additions and 383 deletions
Download patch file Download diff file Expand all files Collapse all files

313
.dirge/skills/jolt-dev/SKILL.md
View file

@ -2,17 +2,104 @@
Jolt development workflow — build, test, special form patterns, Janet gotchas
# Jolt Development
## Build & Test
```bash
cd /Users/yogthos/src/jolt
jpm build # produces build/jolt
jpm test # runs all tests
janet test/foo.janet # run a single test file from project root
git submodule update --init # pulls vendor/sci (required for SCI bootstrap)
jpm build # produces build/jolt
jpm test # runs all tests (9 suites + SCI load)
janet test/foo.janet # run a single test file from project root
```
## Persistent Data Structures
`:mutable?` compile flag in `src/jolt/api.janet` `init`: persistent Clojure data structures loaded by default. Pass `{:mutable? true}` to use Janet native mutable tuples/tables:
```janet
(def ctx (init)) # persistent vectors, maps
(def ctx (init {:mutable? true})) # Janet tuples/structs
```
### Loading .clj source into context
```janet
(def s (slurp "src/jolt/clojure/lang/persistent_vector.clj"))
(var cur s)
(while (> (length (string/trim cur)) 0)
(def [form rest] (parse-next cur))
(set cur rest)
(when (not (nil? form))
(eval-form ctx @{} form)))
;; Then swap bindings:
(let [core-ns (ctx-find-ns ctx "clojure.core")
pv-ns (ctx-find-ns ctx "jolt.lang.persistent-vector")]
(ns-intern core-ns "vec" (var-get (ns-find pv-ns "vector")))
(ns-intern core-ns "vector" (var-get (ns-find pv-ns "vector")))
(ns-intern core-ns "vector?" (var-get (ns-find pv-ns "vector?"))))
```
### deftype table gotchas
- `struct?` returns false for Janet tables, but deftype creates tables. Use `(get val :jolt/deftype)` instead of `(and (struct? val) (get val :jolt/deftype))` for instance? checks.
- `.` special form strips leading `-` from member names before looking up field on deftype instances (e.g., `(.-cnt pv)` → `(get target :cnt)`).
- Default function application path handles `.-field obj` accessor syntax directly.
### Prototype-chain binding lookup
`resolve-sym` needs `binding-get` that walks Janet table prototype chain for nested `let`/`fn` bindings. Two-stage lookup: direct `get` first, then prototype walk:
```janet
(defn- binding-get [bindings name]
(var result :jolt/not-found)
(var t bindings)
(while (not (nil? t))
(when (in t name) (set result (in t name)) (break))
(set t (table/getproto t)))
result)
```
## Macro Patterns
### and/or macros
```janet
;; (and x y) → (let* [and__x x] (if and__x (and y) and__x))
;; (or x y) → (let* [or__x x] (if or__x or__x (or y)))
```
Registered as macros in core-macro-names and bindings.
### loop macro
Expands `(loop [bindings] body...)` → `(loop* [bindings] body...)`. Registered as macro.
### defn multi-arity
Vectors are Janet tuples, not arrays. Use `indexed?` not `array?` for arg pattern matching:
```janet
(if (and (> (length rest) 0) (array? (first rest)) (indexed? (first (first rest))))
;; multi-arity: (defn name ([args] body)...)
...
;; single-arity: (defn name [args] body...)
...)
```
### defn- fix
`defn-` expands to `(def name (fn* ...))` not a reference to the `defn` symbol:
```janet
(defn core-defn- [fn-name & rest]
;; Same logic as defn, emits (def fn-name (fn* ...))
...)
```
### defrecord macro
Builds key-value pairs at macro-expansion time (not eval time — `interleave` doesn't work at eval time in Janet):
```janet
(var kvs @[])
(each f fields-vec
(array/push kvs (keyword (f :name)))
(array/push kvs f))
(def map-expr @[{:jolt/type :symbol :ns nil :name "array-map"} ;kvs])
```
## SCI Bootstrap
SCI added as git submodule at `vendor/sci`. 317/317 forms from 9 core files load with 0 failures in order: macros→protocols→types→unrestrict→vars→lang→utils→namespaces→core. 46 namespaces populated. SCI's `eval-string` is replaced with Jolt-native implementation that delegates to Jolt's reader/evaluator, bypassing SCI's internal pipeline (interpreter, parser, analyzer, opts).
Internal SCI namespaces need edamame/tools.reader stubs. See `test/test-load-sci.janet` for the canonical load test.
## Special Form Checklist
To add a new special form to the evaluator:
@ -28,200 +115,48 @@ The match arm receives `ctx`, `bindings`, and `form` (the full list). Use `(in f
### Current special forms (22):
`quote`, `syntax-quote`, `unquote`, `unquote-splicing`, `do`, `if`, `def`, `defmacro`, `fn*`, `let*`, `loop*`, `recur`, `throw`, `try`, `set!`, `var`, `locking`, `instance?`, `defmulti`, `defmethod`, `deftype`, `new`, `.`
### defmacro details
- Supports optional docstring: `(defmacro name [args] body)` or `(defmacro name "doc" [args] body)`
- Implementation: `(tuple/slice form 2)` → check if first is string → adjust args-form and body start
- Implicit `&env` binding: `(put new-bindings "&env" @{})` — table, not struct (nil-safe)
- Uses `parse-arg-names` for `& rest` arg handling
### defmulti / defmethod
- `defmulti` stores the methods table on the **var** via `(put v :jolt/methods methods)`, NOT on the function
- Janet `put` on a function value fails silently
- `defmethod` retrieves methods via `(get mm-var :jolt/methods)` using `resolve-var` to get the var first
### set!
- If var doesn't exist, auto-creates it (`ns-intern`) rather than erroring
- Needed for sci's `(set! *warn-on-reflection* true)`
### defmethod (auto-create)
- If the multimethod var doesn't exist yet, auto-creates it with a dummy fn and empty methods table
- This allows sci's `defmethod print-method` to work before `defmulti print-method` is defined
### deftype
- Handles `^:meta` on type name via `with-meta` pattern
- Fields vector handles `^:meta` annotations and `^Type` hints — extracts actual symbol name
- Produces a table with `:jolt/deftype "ns.TypeName"` and keyword-keyed fields
## Janet Gotchas
- `var` declares mutable locals that can be `set` later; `def`/`let` are immutable
- `let` cannot bind to `nil` — use `(var x nil)` instead of `(let [x nil] ...)`
- `(get table key)` needs 2 args minimum — for single-arg checks use `(table :key)`
- Functions are not tables — `(put fn :prop val)` fails. Stash properties on vars
- `match` is Janet's pattern matching — no `case` or `cond` needed for simple dispatch
- Janet structs silently **drop entries with nil values**: `(struct ;[:x nil :y 1])` → `{:y 1}`. Use `@{}` mutable tables when nil-valued entries are needed (e.g., `&env` binding `@{}` for macro bodies)
- `match` with string patterns returns **nil** (not error) when no pattern matches. Used in `eval-list` to handle non-symbol heads cleanly — keyword heads like `:ns` fall through to default function application
- `break` in `while` does NOT return a value in Janet. Use `(var done nil)` + `(set done val)` + check pattern instead
- Janet `#{}` set literals can cause parse issues in some contexts — use `@[]` as fallback
- Janet `(put table key nil)` silently drops the key. Use `:jolt/nil` sentinel via `bind-put` helper, unwrapped in `resolve-sym`
- `struct?` returns false for tables — deftype creates tables, use `(get val :key)` for field access
- **Bit operations**: use Janet built-ins `blshift`/`brshift`/`brushift`/`bnot`/`bor`/`band`/`bxor` (not `lshift`/`rshift`/etc.)
- Janet `mod` returns float, not integer
- `#{}` set literals can cause parse issues — use `@[]` as fallback
- `(first struct)` calls `:jolt/type` method — use `(get struct :key)` instead of positional access
- Janet `(struct ;[:x nil])` silently drops the nil entry — the map becomes `{}`. Use `@{}` tables for nil-safe entries
- `(length struct)` counts key-value pairs, not keys. Use `(length (keys struct))` for key count
- **`(last string)` returns nil** — `last` works only on indexed types (tuple, array). For strings use `(s (- (length s) 1))` or `(string/slice s (- (length s) 1))`
- **`(set [a b] tuple)` doesn't work** — Janet's `set` doesn't support destructuring. Use `(tuple 0)` / `(tuple 1)` or explicit individual assignments
- **`(get table key)` does NOT walk prototypes** set by `table/setproto`. Use a custom walker: `(var t bindings) (while t (when (in t key) (set result (in t key)) (break)) (set t (table/getproto t)))`. `(in table key)` DOES walk prototypes.
- **`return` from loop inside `fn` body doesn't work** — use mutable `var result` + `break` pattern instead.
- **Bit-shift functions**: `blshift` (left), `brshift` (right), `brushift` (unsigned right). NOT `lshift`/`rshift`/`urshift`. `brushift` on negative numbers errors.
- **`(hash x)`** built-in works on strings, keywords, numbers, tuples, structs, tables. Used directly as core `hash` binding.
### unwrap-meta-name helper
Recursively unwraps `(with-meta sym meta)` forms to extract the underlying symbol. Used in `def`, `ns`, `deftype`, `defmethod` to handle metadata-wrapped names:
```janet
(defn- unwrap-meta-name [form]
(if (and (array? form) (> (length form) 0)
(struct? (in form 0))
(= :symbol ((in form 0) :jolt/type))
(= "with-meta" ((in form 0) :name)))
(unwrap-meta-name (in form 1))
form))
```
Recursively unwraps `(with-meta sym meta)` forms to extract the underlying symbol. Used in `def`, `ns`, `deftype`, `defmethod` to handle metadata-wrapped names.
### Reader map k/v handling
The map reader must handle three special value types in both key and value positions:
- `:jolt/skip` — discarded form (comment, `#_`, nil `#?(:cljs ...)`): skip the K/V pair entirely
- `:jolt/splice` — `#?@` splicing: concat items into the kvs array. If splice has 0 items (nil `:cljs` branch), don't push the key
- `:jolt/skip` — discarded form: skip the K/V pair entirely
- `:jolt/splice` — `#?@` splicing: concat items into the kvs array
### `#?(:cljs X)` returns nil → `:jolt/skip`
The non-splicing `#?` reader now returns `{:jolt/type :jolt/skip}` for nil results (e.g., `#?(:cljs X)` on CLJ). This prevents orphaned nil keys/values in maps and lists. `parse-next` and `parse-string` skip past skip markers to return the next real form.
The non-splicing `#?` reader returns `{:jolt/type :jolt/skip}` for nil results (e.g., `#?(:cljs X)` on CLJ). This prevents orphaned nil keys/values. `parse-next` and `parse-string` skip past skip markers to return the next real form.
### deftype →TypeName constructor
`deftype` now interns both the type name and `->TypeName` arrow constructor (Clojure convention). The dot-suffix constructor check uses `(sym-name (- (length sym-name) 1))` instead of broken `(last sym-name)`.
Closing delimiters `)`, `]`, `}` in `read-form` produce explicit errors (no fallthrough to `read-symbol`).
### fn* and defmacro namespace capture
fn* and defmacro must capture defining namespace at definition time and restore it during body evaluation:
```
(let [defining-ns (ctx-current-ns ctx)]
...closure body...
(def saved-ns (ctx-current-ns ctx))
(ctx-set-current-ns ctx defining-ns)
...eval body...
(ctx-set-current-ns ctx saved-ns))
```
Applies to single-arity, multi-arity fn*, and defmacro. Without this, closures evaluate in whatever ns happens to be current at call time.
## Core Bindings (145+)
### Persistent data structures
- ClojureScript defines persistent vector as deftype in `cljs/core.cljc`: `PersistentVector` (cnt, shift, root, tail, _meta), `VectorNode` (arr). 32-way branching trie with tail optimization.
- Mutable arrays via `object-array`, `int-array`, `aset`, `aget`, `aclone`, `alength`, `to-array`. Bit ops: `bit-and/or/xor/not`, `bit-shift-left/right`, `unsigned-bit-shift-right`.
- `int` coercion via `math/trunc`. `unchecked-inc/dec/add/subtract` for fast path.
- `.clj` source files in `src/jolt/clojure/lang/`. Loaded by Jolt reader/evaluator at init.
Key bindings in `core-bindings` map (in `src/jolt/core.janet`):
- Predicates, math, collections, sequences, higher-order functions
- Macros: `when`, `when-not`, `if-let`, `when-let`, `if-some`, `when-some`, `doto`, `and`, `or`, `defn`, `defn-`, `fn`, `let`, `loop`, `defrecord`, `defprotocol`, `declare`, `comment`
- Array primitives: `alength`, `aget`, `aset`, `aclone`, `object-array`, `int-array`, `to-array`
- Bit ops: `bit-and`, `bit-or`, `bit-xor`, `bit-not`, `bit-shift-left`, `bit-shift-right`, `unsigned-bit-shift-right`
- Unchecked math: `int`, `unchecked-inc`, `unchecked-dec`, `unchecked-add`, `unchecked-subtract`
- `hash` delegates to Janet built-in `hash`
- `name` returns name string of keyword/symbol/string
- `namespace` returns namespace of keyword/symbol or nil
### instance? and set! for deftype
- `instance?` checks `:jolt/deftype` tag. Matches by suffix: `(instance? PersistentVector x)` → check if `(x :jolt/deftype)` ends with `"PersistentVector"`.
- `set!` handles `(set! (. obj -field) val)` for deftype field mutation. Strips leading `-`, converts to keyword, `(put obj field-key val)`.
### and / or macros
```
(and x y) → (let* [and__x x] (if and__x (and y) and__x))
(or x y) → (let* [or__x x] (if or__x or__x (or y)))
```
Registered as macros in `core-macro-names`.
### defrecord macro
Builds key-value pairs at macro expansion time using `array-map`:
```janet
(var kvs @[])
(each f fields-vec
(array/push kvs (keyword (f :name)))
(array/push kvs f))
(def map-expr @[{:jolt/type :symbol :ns nil :name "array-map"} ;kvs])
```
## Comment Handling
Comments `;` in `read-form` return `{:jolt/type :jolt/skip}` sentinel:
```janet
(= c 59) # ;
[{:jolt/type :jolt/skip} line-end])
```
`parse-next` and `parse-string` both skip over `:jolt/skip` results:
- `parse-next` uses inner `parse-next-loop` that recurses on skip
- `parse-string` recurses on skip to return next non-comment form
`read-map` checks both key AND value positions for `:jolt/skip` to skip `#_`-discarded entries.
Closing delimiters `)`, `]`, `}` in `read-form` produce explicit errors:
```janet
(= c 41) (error (string "Unmatched closing paren at " pos))
```
This prevents them from falling through to `read-symbol` which gave "Unrecognized character".
## `unwrap-meta-name` Utility
Recursively unwraps `(with-meta sym meta)` forms to extract the underlying symbol:
```janet
(defn- unwrap-meta-name [form]
(if (and (array? form) (> (length form) 0)
(struct? (in form 0))
(= :symbol ((in form 0) :jolt/type))
(= "with-meta" ((in form 0) :name)))
(unwrap-meta-name (in form 1))
form))
```
Used in: `def`, `ns` (ns name), `deftype` (type name + field names), `defmethod` (arg names).
Replaced duplicated `with-meta` unwrapping code in each of these forms.
## Bootstrap Patterns
### Class-name resolution in resolve-sym
When a simple symbol (unqualified) isn't found in current ns or clojure.core, checks for dotted class-name pattern:
`Foo.Bar.Baz` → finds last dot → ns "Foo.Bar", name "Baz" → tries ns-resolve.
This resolves symbols like `IVar` that are interned in `sci.impl.vars` but referred to unqualified from `sci.lang`.
### Reader conditionals
- `#?(:clj expr :cljs expr2)` — resolved at read time by `read-reader-conditional`
- `#?@(:clj expr :cljs expr2)` — splicing variant, wraps resolved items in `{:jolt/type :jolt/splice :items ...}`
- List/vector/set readers check for splice and flatten items
- `#_` — discard reader macro, reads next form and returns it as position only
### Core macros (in core.janet)
- `core-macro-names` returns `@{"when" true "defn" true "declare" true "defprotocol" true "extend-type" true "extend-protocol" true "extend" true "reify" true "fn" true "proxy" true "definterface" true "comment" true "defn-" true}` — a table
- `init-core!` calls `(get (core-macro-names) name)` to check, then `(put v :macro true)`
- Order matters: macro functions must be defined BEFORE `core-bindings` map references them
### Core stubs for sci bootstrap
- `core-derive`, `core-isa?`, `core-ancestors`, `core-descendants` — minimal hierarchy
- `core-Object`, `core-Thread`, `core-ThreadLocal`, `core-IllegalStateException` — JVM class stubs
- `core-volatile!`, `core-vswap!`, `core-vreset!` — volatile (atom-like table with :val key)
- `core-defprotocol` emits `(do (def name @{}) (def method fn) ...)` — macro, returns do form
- `core-extend-type`, `core-extend-protocol`, `core-extend`, `core-reify`, `core-satisfies?`, `core-extends?`, `core-implements?`, `core-type->str` — protocol stubs
### Namespace handling
- `ns` form handles `^:meta` on ns name via `with-meta` pattern
- `def` form also handles `^:meta` on def name (extracts name-sym from `(with-meta Name meta)`)
- `require` clause in `ns` wraps each spec in `(when s ...)` for nil-safety
- `resolve-var` falls back to checking clojure.core namespace if var not found in current ns
### Bootstrap loading order
```
sci.impl.macros (4/4 ok)
sci.impl.protocols (15/17 ok)
sci.impl.utils (39/47 ok)
sci.impl.types (22/27 ok)
sci.impl.unrestrict (2/2 ok)
sci.impl.vars (28/28 ok — comment block parsed as skip)
sci.lang (10/10 ok — IVar via class-name resolve)
sci.ctx-store (6/6 ok)
sci.impl.namespaces (93/98 ok — parse crash at unmatched brace)
sci.core (60/69 ok — namespaces/*1/*2/*3/*e fail)
```
All .cljc files. #?(:clj ...) resolved at read time. #?(:cljs ...) returns nil.
### parse-arg-names
- Handles `& rest` args AND nested destructuring vectors
- When an arg is a vector (not a symbol), recurses to extract nested symbol names
### core-macro-names
Table `@{"when" true "defn" true ...}` — maps symbol name → `true` for all macro bindings. `init-core!` checks this to set `:macro true` on vars.
## Project Structure
@ -230,19 +165,13 @@ src/jolt/
types.janet — Var, Namespace, Context, symbol helpers
reader.janet — recursive descent parser for Clojure syntax
evaluator.janet — tree-walking interpreter (eval-form, eval-list, syntax-quote*)
core.janet — 95+ clojure.core functions (map, filter, reduce, etc.)
core.janet — 145+ clojure.core functions and macros
api.janet — public API: init, eval-string, eval-string*
main.janet — REPL entry point with (defn main [&])
main.janet — REPL entry point
clojure/lang/
persistent_vector.clj — 32-way branching trie vector (17 forms)
persistent_hash_map.clj — HAMT hash map (24 forms, WIP)
test/
evaluator-test.janet — special form tests (22 forms tested)
reader-test.janet — parser tests (includes #?, #?@, #_)
core-test.janet — core library tests
macro-test.janet — syntax-quote and macro tests
namespace-test.janet — ns, require, in-ns tests
types-test.janet — Var and Namespace tests
api-test.janet — public API tests
bootstrap-test.janet — loads sci.impl.macros (deftime, usetime, ?)
test-load-sci.janet — loads all sci files, counts ok/fail
test-eval.janet — end-to-end sci.core/eval-string test
test/ — 9 test suites + SCI load test
vendor/sci/ — SCI submodule
```