Merge pull request #116 from jolt-lang/refactor-phase5de-seed-overlay-docs

Refactor phase 5d/5e: seed↔overlay registry + boundary docs (jolt-bvek)

docs/seed-overlay-registry.md

@@ -0,0 +1,80 @@
+# Seed ↔ Overlay Registry
+
+Jolt is "Clojure on Janet": a shrinking **Janet seed** (`src/jolt/*.janet`)
+hosts a **Clojure overlay** (`jolt-core/clojure/core/NN-*.clj`). Both define
+`clojure.core`-facing functions, and for a handful of names *both* tiers carry a
+definition. Which copy is authoritative has been tribal knowledge. This document
+is the single source of truth; `test/unit/seed-overlay-registry-test.janet` is a
+build-time drift check that fails if reality diverges from what is written here.
+
+## The registration mechanism
+
+Seed core functions are named with a `core-` prefix (`core-into`, `core-conj`,
+`core-transduce`) and registered into the `clojure.core` namespace by the
+`core-bindings` table in `src/jolt/core.janet`. Each entry maps a **public
+Clojure name** (the string key) to a seed function value:
+
+```janet
+(def- core-bindings
+  @{"into"   core-into
+    "reduce" core-reduce
+    ...})
+```
+
+`init-core!` (`src/jolt/core.janet`) interns every pair into `clojure.core`.
+The overlay tiers load afterwards (`api.janet`: 00-syntax, 00-kernel, 10-seq,
+20-coll, 25-sorted, 30-macros, 40-lazy, 50-io). When an overlay tier `(defn X …)`
+for a name that `core-bindings` already registered, the **overlay def shadows the
+seed binding** — the seed `core-X` then survives only if some other seed code
+still calls it directly.
+
+So a name's *home* is determined by two facts:
+
+1. is it a key in `core-bindings`? (registered ⇒ the seed `core-X` is reachable)
+2. does an overlay tier `(defn X …)`? (defined ⇒ the overlay copy shadows)
+
+## Dispatch-only seed helpers: the `__` prefix
+
+Seed functions that are **not** public Clojure vars but must be reachable by
+name from compiled/overlay code (compiler hooks, macro-expansion targets) are
+registered under a `__`-prefixed key — e.g. `"__sq1"`, `"__write"`,
+`"__bit-and"`, `"__jdbc-conn-raw"`. The `__` prefix is unreadable as a
+user-level symbol, so these never collide with or masquerade as public API. When
+you add a dispatch-only hook, give it a `__` key; do not register it under a bare
+name.
+
+## Dispatch twins
+
+A **twin** is a name with *both* a seed `core-X` defn and an overlay `(defn X …)`.
+There are exactly five. Each seed site carries a greppable `SEED-TWIN:` comment.
+
+| name          | overlay (authoritative public)        | seed copy (`core-X`)                | registered? | role of the seed copy |
+|---------------|----------------------------------------|--------------------------------------|-------------|------------------------|
+| `char?`       | `20-coll.clj` `char?`                  | `core_types.janet` `core-char?`      | no          | internal type dispatch |
+| `sorted-map?` | `25-sorted.clj` `sorted-map?`          | `core_types.janet` `core-sorted-map?`| no          | internal dispatch (sorted-op) |
+| `sorted-set?` | `25-sorted.clj` `sorted-set?`          | `core_types.janet` `core-sorted-set?`| no          | internal dispatch |
+| `sorted?`     | `25-sorted.clj` `sorted?`              | `core_types.janet` `core-sorted?`    | no          | internal dispatch |
+| `transduce`   | `20-coll.clj` `transduce`              | `core_coll.janet` `core-transduce`   | no          | internal helper for `core-into` only |
+
+None of the five is registered in `core-bindings`: the overlay copy is the public
+one, and the seed copy is reached only by other *seed* code (so editing the seed
+copy alone will not change what user code sees — change both, or move the logic).
+
+## The surprising asymmetry: `into` vs `transduce`
+
+`into` and `reduce` are **seed-public**: registered in `core-bindings`, and the
+overlay deliberately does *not* redefine them (they sit on the perf wall — see
+the "into stays in the seed" note in `20-coll.clj`). `transduce`, by contrast, is
+**overlay-public**: the overlay `transduce` is the real one, and `core-transduce`
+remains only because `core-into` calls it directly. So two functions that read as
+a matched pair have opposite homes. That asymmetry is intentional and is the
+reason this registry exists.
+
+## Drift check
+
+`test/unit/seed-overlay-registry-test.janet` recomputes the twin set from source
+(names with both a seed `core-X` defn and an overlay `defn X`) and asserts it
+equals the five above. It also asserts none of the twins is registered in
+`core-bindings`, and that every non-`__` `core-bindings` key is a plausible
+public name (no accidental `__`-less dispatch helper). If you add, remove, or
+re-home a twin, update this table and that test together.

src/jolt/core_coll.janet

@@ -1,5 +1,11 @@
 # Jolt Core — collections, transducers, seqs, HOFs, constructors
 # Extracted from core.janet (jolt-nma8, phase 2b split).
+#
+# REP vs API: this file holds the Clojure-facing collection ops and dispatches
+# on `:jolt/type` over the internal persistent structures, whose representations
+# live elsewhere: persistent vector → pv.janet, list → plist.janet, hash map →
+# phm.janet, set → phs.janet, lazy seq → lazyseq.janet. Grep a structure's file
+# header for the primitive (pv-*/pl-*/phm-*/phs-*/ls-*) it exposes.
 
 (use ./types)
 (use ./phm)
@@ -535,6 +541,10 @@
   [rf init coll]
   (reduce-with-reduced rf init coll))
 
+# SEED-TWIN: transduce is overlay-public (jolt-core/clojure/core/20-coll.clj);
+# this seed copy is NOT registered in core-bindings. It survives only as the
+# helper core-into calls below — user `transduce` resolves to the overlay. The
+# asymmetry with `into` (seed-public) is intentional; docs/seed-overlay-registry.md.
 (defn core-transduce
   "(transduce xform f coll) or (transduce xform f init coll)."
   [xform f & rest]

src/jolt/core_types.janet

@@ -50,6 +50,10 @@
 # Sorted-coll tag checks + entries view, defined this early because canon-key,
 # empty?, and jolt-equal? (all below) need them. The sorted-coll SEMANTICS are
 # pure Clojure (core/25-sorted.clj); see the dispatch section further down.
+# SEED-TWIN: sorted-map?/sorted-set?/sorted? also live in the overlay
+# (jolt-core/clojure/core/25-sorted.clj); the overlay copies are the public ones
+# (NOT in core-bindings). These seed copies exist only for earlier-tier seed
+# dispatch. Change both copies together — docs/seed-overlay-registry.md.
 (defn core-sorted-map? [x] (and (table? x) (= :jolt/sorted-map (x :jolt/type))))
 (defn core-sorted-set? [x] (and (table? x) (= :jolt/sorted-set (x :jolt/type))))
 (defn core-sorted? [x] (or (core-sorted-map? x) (core-sorted-set? x)))
@@ -194,6 +198,9 @@
 # Predicates
 # ============================================================
 
+# SEED-TWIN: char? is also defined in the overlay (jolt-core/clojure/core/
+# 20-coll.clj) and the overlay copy is the public one (NOT in core-bindings).
+# This seed copy is internal type dispatch only. docs/seed-overlay-registry.md.
 (defn core-char? [x] (and (struct? x) (= :jolt/char (x :jolt/type))))
 (defn char-code [c] (c :ch))
 (defn char->string [c] (string/from-bytes (c :ch)))

src/jolt/lazyseq.janet

@@ -3,7 +3,11 @@
 # Model: a thunk returns nil (empty) or a [first-val, rest-thunk] pair; each
 # step produces one element + a thunk for the rest. Supports self-referencing
 # sequences like fib-seq. Self-contained (janet builtins only) — the Clojure
-# seq layer (core.janet) and the interpreter build on these primitives.
+# seq layer (core_coll.janet) and the interpreter build on these primitives.
+#
+# REP vs API: this file is ONLY the lazy-cell representation (ls-* primitives).
+# The Clojure-facing seq ops (first/rest/seq/cons/count dispatch, realization)
+# live in core_coll.janet, branching on `:jolt/type :jolt/lazy-seq`.
 #
 # Extracted from phm.janet (jolt-bvek): a lazy sequence has nothing to do with
 # hash maps; both were tagged tables, which is why they shared a file.

src/jolt/phm.janet

@@ -5,6 +5,12 @@
 # ~12-entry linear scan per get (the jolt-s3y map-read regression). The bucket
 # count is derived from (length (m :buckets)), so marshaled images from before
 # this change keep working.
+#
+# REP vs API: this file is ONLY the hash-map representation (phm-* primitives).
+# The Clojure-facing map ops (assoc/dissoc/get/conj/count/seq dispatch, nil-key
+# handling, merge) live in core_coll.janet / core_types.janet, which recognize
+# phm by its `:jolt/deftype` string. PersistentHashSet is layered on top in
+# phs.janet; LazySeq (historically here) now lives in lazyseq.janet.
 
 (def- initial-buckets 8)
 

src/jolt/phs.janet

@@ -1,6 +1,11 @@
 # PersistentHashSet — a set backed by a PersistentHashMap (members are keys
 # mapped to true). Extracted from phm.janet (jolt-bvek) so phm.janet is purely
 # the hash map; the set is a thin layer over it.
+#
+# REP vs API: this file is ONLY the set representation (phs-* primitives). The
+# Clojure-facing set ops (conj/disj/contains?/count/seq dispatch, set-as-fn
+# membership) live in core_coll.janet / core_types.janet, branching on
+# `:jolt/type :jolt/set`.
 
 (use ./phm)
 

src/jolt/plist.janet

@@ -12,6 +12,10 @@
 #
 # `:rest` may be a plain array/tuple so `(conj some-list x)` needn't copy the
 # original list — the node just references it. pl->array materializes the chain.
+#
+# REP vs API: this file is ONLY the cons-list representation (pl-* primitives).
+# The Clojure-facing list/seq ops (conj/cons/first/rest/count/seq dispatch) live
+# in core_coll.janet, which branches on `:jolt/type :jolt/plist`.
 
 (defn plist? [x]
   (and (table? x) (= :jolt/plist (get x :jolt/type))))

src/jolt/pv.janet

@@ -11,6 +11,11 @@
 #     :tail  tail}      a tuple of up to 32 trailing elements (append fast-path)
 #
 # Trie nodes are immutable tuples so unchanged subtrees are shared by identity.
+#
+# REP vs API: this file is ONLY the trie representation (pv-* primitives). The
+# Clojure-facing vector ops (nth/conj/assoc/subvec, the count/seq/first/rest
+# dispatch, and tuple↔pvec polymorphism) live in core_coll.janet /
+# core_types.janet, which dispatch on `:jolt/type :jolt/pvec`.
 
 (def- bits 5)
 (def- width 32)         # 2^bits

test/unit/seed-overlay-registry-test.janet

@@ -0,0 +1,104 @@
+# Drift check for the seed ↔ overlay boundary (refactor phase 5d).
+#
+# Recomputes the dispatch-twin set from source and asserts it matches what
+# docs/seed-overlay-registry.md documents. A twin is a name with BOTH a seed
+# `core-X` defn (src/jolt/*.janet) and an overlay `(defn X …)`
+# (jolt-core/clojure/core/*.clj). If you add, remove, or re-home a twin, update
+# the doc table and the EXPECTED-TWINS set below together.
+#
+# This is source analysis, not Clojure eval — a plain Janet test (no harness).
+
+(def repo-root
+  # test/unit/<this> -> repo root is two levels up
+  (let [d (os/cwd)] d))
+
+(defn- slurp-safe [path]
+  (try (slurp path) ([_] nil)))
+
+(defn- files-with-ext [dir ext]
+  (def acc @[])
+  (each name (os/dir dir)
+    (def full (string dir "/" name))
+    (case (os/stat full :mode)
+      :directory (array/concat acc (files-with-ext full ext))
+      :file (when (string/has-suffix? ext name) (array/push acc full))))
+  acc)
+
+# Collect bare names from `(defn NAME` / `(defn- NAME`, optionally stripping a
+# `prefix` (e.g. "core-") and keeping only those that had the prefix.
+(defn- defn-names [src prefix]
+  (def names @{})
+  (def pat (peg/compile
+             ~{:sym (capture (some (+ (range "az" "AZ" "09") (set "*?!<>=+/.&_-"))))
+               :defn (* "(defn" (? "-") (some (set " \t")) :sym)
+               :main (some (+ :defn 1))}))
+  (each m (or (peg/match pat src) @[])
+    (when (string? m)
+      (if prefix
+        (when (string/has-prefix? prefix m)
+          (put names (string/slice m (length prefix)) true))
+        (put names m true))))
+  names)
+
+(defn- merge-into [dst src] (eachk k src (put dst k true)) dst)
+
+# --- seed core-X names across src/jolt/*.janet ---
+(def seed-names @{})
+(each f (files-with-ext (string repo-root "/src/jolt") ".janet")
+  (merge-into seed-names (defn-names (slurp f) "core-")))
+
+# --- overlay public defns across jolt-core/clojure/core/*.clj ---
+(def overlay-names @{})
+(each f (files-with-ext (string repo-root "/jolt-core/clojure/core") ".clj")
+  (merge-into overlay-names (defn-names (slurp f) nil)))
+
+# --- twins = intersection ---
+(def twins @{})
+(eachk k seed-names (when (get overlay-names k) (put twins k true)))
+
+(def EXPECTED-TWINS
+  {"char?" true "sorted?" true "sorted-map?" true "sorted-set?" true
+   "transduce" true})
+
+(defn- keyset [t] (sort (keys t)))
+
+(unless (deep= (keyset twins) (keyset EXPECTED-TWINS))
+  (error (string "seed↔overlay twin drift!\n"
+                 "  computed: " (string/join (keyset twins) ", ") "\n"
+                 "  expected: " (string/join (keyset EXPECTED-TWINS) ", ") "\n"
+                 "  Update docs/seed-overlay-registry.md and EXPECTED-TWINS together.")))
+
+# --- core-bindings registered keys (the public/dispatch registration table) ---
+(def core-src (slurp (string repo-root "/src/jolt/core.janet")))
+(def bind-start (string/find "@{" core-src (string/find "core-bindings" core-src)))
+# brace-match from the `@{` to its close
+(defn- match-brace [src open]
+  (var depth 0) (var i open) (var end nil)
+  (while (and (< i (length src)) (nil? end))
+    (case (src i)
+      (chr "{") (++ depth)
+      (chr "}") (do (-- depth) (when (= depth 0) (set end i))))
+    (++ i))
+  end)
+(def bind-end (match-brace core-src (+ bind-start 1)))
+(def bind-region (string/slice core-src bind-start (+ bind-end 1)))
+(def registered @{})
+(each m (or (peg/match ~(some (+ (* "\"" (capture (some (if-not "\"" 1))) "\"") 1)) bind-region) @[])
+  (put registered m true))
+
+# Twins must NOT be registered — the overlay copy shadows; the seed copy is
+# internal-only. A twin sneaking into core-bindings means the seed copy is
+# masquerading as the public binding.
+(eachk t EXPECTED-TWINS
+  (when (get registered t)
+    (error (string "twin '" t "' is registered in core-bindings — it should be "
+                   "overlay-public only (see docs/seed-overlay-registry.md)"))))
+
+# The seed-public anchors must stay registered (guards the into/transduce
+# asymmetry the registry documents).
+(each anchor ["into" "reduce"]
+  (unless (get registered anchor)
+    (error (string "seed-public anchor '" anchor "' missing from core-bindings"))))
+
+(print "seed↔overlay registry: " (length (keys twins)) " twins, "
+       (length (keys registered)) " registered bindings — OK")