Docs: Chez-only, drop the Janet-era references and obsolete migration notes
Bring the docs in line with the actual implementation now that Chez is the sole substrate. Deleted the migration/spike/handoff artifacts that only documented the Janet era or the port effort: the port plan, phase-0 and foundational-runtime spike writeups (+ the stray root-level copy), the self-hosting design notes, the architecture-refactor plan, and spike/chez/RESULTS.md. Rewrote the current reference docs against the Chez facts: building-and-deps and tools-deps (no jpm/build step — bin/joltc off the checked-in seed, deps via jolt.deps into ~/.jolt/gitlibs), libraries (SQLite is built-in jdbc.core over libsqlite3, not a Janet driver), the conformance/spec test-flow docs (the Chez corpus runner + certify, no .janet harnesses), and the transient / type-hint / seed-overlay design notes (Chez representations: mutable transients, flat copy-on-write vectors, HAMT maps, the seed/overlay twin). Fixed the README collections line (vectors aren't 32-way tries) and added the ffi/transient gate targets. rfc 0001's numerics open-question is resolved (the Scheme tower). Renamed the built-in HTTP adapter to jolt.http.server only (dropped the ring-janet.adapter alias — a Janet-era name).
This commit is contained in:
Yogthos
parent
fe3fdf6b9c
commit
45876998ad
28 changed files with 253 additions and 2012 deletions
|
|
@ -170,9 +170,9 @@ Signature(s), since-version
|
|||
## Open questions
|
||||
|
||||
1. Numerics: the reference has longs/doubles/ratios/BigInt with promotion
|
||||
rules; CLJS has JS numbers; jolt has Janet numbers. Likely answer: specify
|
||||
an integer/float core with a host-numeric-tower extension point — needs
|
||||
its own design note in §4.
|
||||
rules; CLJS has JS numbers. Resolved: jolt carries the Scheme numeric tower
|
||||
(exact integers/bignums, exact ratios, flonum doubles), matching the
|
||||
reference's tower — see the numerics note in §4.
|
||||
2. Where do `*print-length*`-style dynamic vars land — host-dependent
|
||||
interface or portable with defaults?
|
||||
3. License/venue if the spec outgrows this repo (likely CC-BY; separate repo
|
||||
|
|
|
|||
|
|
@ -1,50 +1,49 @@
|
|||
# RFC 0003: Transients — semantics and why they live in the Janet seed
|
||||
# RFC 0003: Transients — semantics and the Chez mutable backing
|
||||
|
||||
Status: accepted (design note)
|
||||
|
||||
This note pins down what transients *are* in Jolt, where their behavior
|
||||
deviates from JVM Clojure and why, and why the transient machinery is part of
|
||||
the irreducible Janet seed rather than a candidate for the core-in-Clojure
|
||||
migration (jolt-tzo). It exists so the kernel-shrink ladder doesn't revisit
|
||||
deviates from JVM Clojure and why, and how the transient machinery is
|
||||
represented in the Chez runtime. It exists so the design doesn't revisit
|
||||
transients every round.
|
||||
|
||||
## What a transient is in Jolt
|
||||
|
||||
A transient is a tagged Janet table wrapping a *native* mutable host value
|
||||
(`core.janet`, "Transients" section):
|
||||
A transient is a Chez record (`jolt-transient`, `host/chez/transients.ss`)
|
||||
wrapping *true mutable* host backing, snapshotted to the immutable collection on
|
||||
`persistent!`. The backing is per kind:
|
||||
|
||||
- transient vector — `@{:jolt/type :jolt/transient :kind :vector :arr ARRAY}`,
|
||||
a Janet array.
|
||||
- transient map — `:kind :map :tbl TABLE`, a Janet table mapping
|
||||
`canon-key(k)` → `@[k v]`. Keying by canonical key keeps collection keys
|
||||
comparing by value across representations (`[1 2]` the pvec and `[1 2]` the
|
||||
tuple are one key), and storing the `@[k v]` pair preserves the *original*
|
||||
key for the rebuilt persistent map.
|
||||
- transient set — `:kind :set :tbl TABLE` mapping `canon-key(x)` → `x`.
|
||||
- transient vector — a growable Scheme vector (a capacity buffer plus a fill
|
||||
count `n`). `conj!`/`pop!` are in-place, amortized O(1); the buffer doubles on
|
||||
growth.
|
||||
- transient map — a Chez hashtable keyed by `key-hash` / `jolt=`
|
||||
(value-equality, nil-safe). Hashing by value keeps collection keys comparing
|
||||
across representations.
|
||||
- transient set — a Chez hashtable of elements.
|
||||
- `cow` — a copy-on-write fallback for anything else (e.g. a sorted coll).
|
||||
|
||||
`transient` accepts pvecs, mutable-build arrays, tuples (reader vectors and
|
||||
map entries — added in the seed-shrink rounds so `(into [] (first {:a 1}))`
|
||||
works through the vector fast path), sets, phms, and untagged struct maps.
|
||||
Sorted collections are rejected, as on the JVM (not editable).
|
||||
`transient` accepts pvecs, pmaps, psets, and the exotic colls handled by the
|
||||
`cow` path. Each kind copies its source into the matching mutable backing once.
|
||||
|
||||
The bang ops (`conj!`, `assoc!`, `dissoc!`, `disj!`, `pop!`) mutate that host
|
||||
value in place and return the transient — O(1) per op (amortized for array
|
||||
push). `persistent!` rebuilds a persistent value from the host value and
|
||||
invalidates the transient (`:jolt/persistent` flag; any further bang op or a
|
||||
second `persistent!` throws "Transient used after persistent! call", matching
|
||||
Clojure's invalidation contract).
|
||||
The bang ops (`conj!`, `assoc!`, `dissoc!`, `disj!`, `pop!`) mutate that backing
|
||||
in place and return the transient — O(1) per op (amortized for the vector push).
|
||||
`persistent!` snapshots a persistent value from the backing (folding the
|
||||
hashtable into a pmap/pset, handing off the buffer as a pvec) and invalidates the
|
||||
transient (the record's active flag clears; any further bang op or a second
|
||||
`persistent!` throws "transient used after persistent!", matching Clojure's
|
||||
invalidation contract).
|
||||
|
||||
Read ops work on an active transient where Clojure supports them: `get`,
|
||||
`contains?`, `count`, and `nth` (vector kind) branch on the transient tag.
|
||||
`contains?`, `count`, and `nth` (vector kind) see through the transient.
|
||||
`seq` on a transient is not supported, as in Clojure.
|
||||
|
||||
## Deviations from JVM Clojure (deliberate)
|
||||
|
||||
**O(n) edges, O(1) middle.** Clojure's `(transient v)` is O(1) — the transient
|
||||
*shares* the persistent trie and marks nodes editable; `persistent!` is O(1)
|
||||
too. Jolt's `transient` copies the source into a native array/table (O(n)) and
|
||||
`persistent!` rebuilds (O(n)). The bang ops in between are native-host O(1),
|
||||
which is *faster* per-op than trie editing. So the asymptotics of the usual
|
||||
too. Jolt's `transient` copies the source into a mutable buffer/hashtable (O(n))
|
||||
and `persistent!` snapshots back (O(n)). The bang ops in between are host-mutable
|
||||
O(1), which is *faster* per-op than trie editing. So the asymptotics of the usual
|
||||
pattern
|
||||
|
||||
(persistent! (reduce conj! (transient []) coll))
|
||||
|
|
@ -53,13 +52,12 @@ are identical (O(n) total either way) with a better constant in the loop and a
|
|||
worse constant at the two edges. The pattern transients exist for — batch
|
||||
construction — is fully served. What is NOT served is transient-editing a
|
||||
*large* collection to change a few keys: that's O(n) in Jolt vs O(log n) in
|
||||
Clojure, because `transient` flattens the pvec trie / phm HAMT into a
|
||||
native array/table and `persistent!` rebuilds them.
|
||||
Clojure, because `transient` copies the source into a growable Scheme vector /
|
||||
Chez hashtable and `persistent!` snapshots it back.
|
||||
|
||||
**No thread-ownership check.** JVM Clojure ≥1.7 also dropped the owner-thread
|
||||
assertion (for fork/join), keeping only "don't use after persistent!", which
|
||||
Jolt enforces. Jolt code is fiber-concurrent; when real OS-thread futures land
|
||||
(jolt-ejx), a transient handed across threads is a data race exactly as in
|
||||
Jolt enforces. A transient handed across threads is a data race exactly as in
|
||||
Clojure — documented, not checked, same as the JVM.
|
||||
|
||||
**`(conj!)` / `(conj! t)` arities** follow Clojure's transducer-era contract:
|
||||
|
|
@ -68,51 +66,43 @@ zero args makes a fresh `(transient [])`, one arg returns it untouched.
|
|||
lenient kvs walk of Jolt's `assoc`.
|
||||
|
||||
**No transient sorted variants** — same as Clojure. One leniency: Clojure
|
||||
throws on `(transient '(1))`, but Jolt's lists are Janet arrays underneath and
|
||||
fall into the mutable-build branch, yielding a transient *vector*. Harmless
|
||||
(the result of `persistent!` is a vector, never silently a list) but
|
||||
non-Clojure; tighten if it ever bites.
|
||||
throws on `(transient '(1))`, but Jolt routes a list through the `cow` fallback
|
||||
path, yielding a transient. Harmless but non-Clojure; tighten if it ever
|
||||
bites.
|
||||
|
||||
## Why transients stay in the Janet seed
|
||||
## Why transients live in the host
|
||||
|
||||
The migration ladder (jolt-tzo) moves anything expressible as *pure Clojure
|
||||
over existing primitives* out of the seed. Transients fail that test on three
|
||||
Transients are part of the value/representation layer in the Chez runtime
|
||||
(`host/chez/transients.ss`), not the portable `clojure.core` overlay, on three
|
||||
grounds:
|
||||
|
||||
1. **They are the mutation kernel.** A transient's entire value is direct
|
||||
mutation of a host array/table. The overlay's only mutation seam is
|
||||
`jolt.host/ref-put!` (a single table-put). Re-expressing `tr-conj!` etc. in
|
||||
Clojure would mean either growing the host surface one-for-one
|
||||
(`host-array-push!`, `host-table-put!`, …, i.e. moving the same code behind
|
||||
more indirection) or simulating mutation over persistent values (defeating
|
||||
the point of transients). Either way the Janet line count moves, it doesn't
|
||||
shrink.
|
||||
mutation of a host buffer/hashtable. The overlay has no mutation seam of its
|
||||
own. Re-expressing the bang ops in Clojure would mean either growing the host
|
||||
surface one-for-one (a host-vector-push, a host-hashtable-put, …, i.e. moving
|
||||
the same code behind more indirection) or simulating mutation over persistent
|
||||
values (defeating the point of transients).
|
||||
|
||||
2. **They sit under the seed's own dispatch.** `conj`/`assoc`/`get`/`count`/
|
||||
`contains?` in the seed branch on the transient tag. Hoisting the transient
|
||||
ops above that dispatch (the hierarchy-port pattern of lazily-resolved
|
||||
overlay vars) would put an interpreted/compiled-Clojure call inside the
|
||||
hottest native paths for no semantic gain — transients have no semantics to
|
||||
*fix* (unlike hierarchy, which had real correctness gaps).
|
||||
2. **They sit under the collection dispatch.** `conj`/`assoc`/`get`/`count`/
|
||||
`contains?` see through a transient. Hoisting the transient ops above that
|
||||
dispatch would put a compiled-Clojure call inside the hottest paths for no
|
||||
semantic gain — transients have no semantics to *fix*.
|
||||
|
||||
3. **The value layer is declared irreducible.** The self-hosting design doc
|
||||
(docs/self-hosting-compiler.md, "The kernel") keeps the value/representation
|
||||
layer — persistent collections and, with them, their mutable scratch
|
||||
counterparts — in the host. Transients are representation, not library.
|
||||
3. **The value layer is the host's job.** The persistent collections and, with
|
||||
them, their mutable scratch counterparts, live in the Chez runtime alongside
|
||||
the value model. Transients are representation, not library.
|
||||
|
||||
What CAN move (and mostly has): anything *derived* — e.g. `into`'s
|
||||
transient-using fast path, or future `update!`-style conveniences — is plain
|
||||
Clojure over `transient`/bang-ops/`persistent!` and belongs in the overlay
|
||||
tiers as ordinary migration batches.
|
||||
What lives in the overlay: anything *derived* — e.g. `into`'s transient-using
|
||||
fast path, or `update!`-style conveniences — is plain Clojure over
|
||||
`transient`/bang-ops/`persistent!`.
|
||||
|
||||
## Future work
|
||||
|
||||
- pvec is already a 32-way trie with structural sharing (pv.janet), so
|
||||
Clojure-style O(1) `transient`/`persistent!` via editable nodes is a real
|
||||
option for vectors — an internal change behind the same surface, not a
|
||||
semantics change. phm is now a HAMT with structural sharing too (jolt-684u),
|
||||
and sorted maps/sets are a red-black tree (jolt-0hbr), so the same editable-
|
||||
node trick is open for those as well — the transient surface here is still the
|
||||
copy-to-native-table flatten.
|
||||
- The persistent map/set are a bitmap HAMT with structural sharing
|
||||
(`host/chez/collections.ss`), so Clojure-style O(1) `transient`/`persistent!`
|
||||
via editable nodes is a real option there — an internal change behind the same
|
||||
surface, not a semantics change. The persistent vector is a flat
|
||||
copy-on-write Scheme vector rather than a trie, so the transient surface for
|
||||
it stays the copy-to-growable-vector path.
|
||||
- `transient?` (Jolt extension, useful in tests) stays; Clojure has no public
|
||||
predicate, so it must not leak into portability-sensitive code.
|
||||
|
|
|
|||
|
|
@ -11,26 +11,19 @@ measured effect, so later work does not relitigate it.
|
|||
## Background: why the lookup carries a guard
|
||||
|
||||
A Jolt map value has several runtime representations (see RFC on collections and
|
||||
`src/jolt/core.janet`): a Janet struct for a small all-scalar-key literal map, a
|
||||
persistent hash map (a table tagged `:jolt/type :jolt/phm`) when a key is a
|
||||
collection or a value is nil, plus sorted maps, transients, and record/deftype
|
||||
instances. A record instance is a Janet table tagged `:jolt/deftype` but, like a
|
||||
struct, it carries no `:jolt/type`, so a raw Janet `(get inst :field)` reads its
|
||||
fields directly.
|
||||
`host/chez/collections.ss`): a persistent hash map (a bitmap HAMT) for the
|
||||
general case, plus sorted maps, transients, and record/deftype instances. A
|
||||
record instance is a Chez record (`jrec`) whose fields are read directly off the
|
||||
record's storage, while a HAMT lookup runs the full `jolt=`/`jolt-hash`-keyed
|
||||
collection path.
|
||||
|
||||
A constant-keyword lookup `(:k m)` compiles to a guarded form:
|
||||
|
||||
```janet
|
||||
(if (get m :jolt/type) (core-get m k) (get m k))
|
||||
```
|
||||
|
||||
The guard is one opcode. A non-nil `:jolt/type` routes phm/sorted/transient/
|
||||
lazy-seq values to `core-get`'s full semantics; everything else (structs,
|
||||
records, nil, scalars) takes the bare Janet `get`, which matches `core-get` for
|
||||
keyword keys. The guard is correct and cheap, but on a struct it is a second
|
||||
`get`: profiling the ray tracer (a naive all-maps program) found keyword lookups
|
||||
are about half of a render, and the guard is the only avoidable part of each
|
||||
one. A bare get is roughly 20ns where the guarded form is roughly 36ns.
|
||||
A constant-keyword lookup `(:k m)` compiles to a guarded form: it inspects the
|
||||
subject's representation and routes a HAMT/sorted/transient/lazy-seq value to the
|
||||
full `jolt-get` semantics, while a record/raw-get-safe value takes the direct
|
||||
field read, which matches `jolt-get` for keyword keys. The guard is correct and
|
||||
cheap, but on a raw-get-safe value it is wasted work: profiling the ray tracer (a
|
||||
naive all-maps program) found keyword lookups are about half of a render, and the
|
||||
guard is the only avoidable part of each one.
|
||||
|
||||
Dropping the guard is only safe when the subject is known to be a plain
|
||||
struct/record rather than a tagged collection. Jolt does not infer that
|
||||
|
|
@ -59,27 +52,27 @@ optimization.
|
|||
## How it flows
|
||||
|
||||
The reader already keeps `^hint` metadata on the binding symbol and is otherwise
|
||||
transparent (`reader.janet`, `meta-form->map`). The change threads that fact to
|
||||
the lookup site:
|
||||
transparent (`host/chez/reader.ss`). The change threads that fact to the lookup
|
||||
site:
|
||||
|
||||
1. The analyzer (`jolt-core/jolt/analyzer.clj`) records a `:struct` hint per
|
||||
local in its env when a param or `let` binding carries `^:struct` or a
|
||||
record-type tag, and attaches `:hint :struct` to that local's `:local` IR
|
||||
node. Resolving a record-type tag uses a new host contract function
|
||||
`record-type?` (`src/jolt/host_iface.janet`), which checks for the `->Name`
|
||||
constructor.
|
||||
2. The back end (`emit-kw-lookup` in `src/jolt/backend.janet`) emits the bare get
|
||||
node. Resolving a record-type tag uses the host contract function
|
||||
`record-type?` (`jolt.host`, backed by `host/chez/host-contract.ss`), which
|
||||
checks for the `->Name` constructor.
|
||||
2. The back end (`jolt-core/jolt/backend_scheme.clj`) emits the direct field read
|
||||
when the lookup subject is a `:local` carrying the hint, and the guarded form
|
||||
otherwise. The unhinted path is byte-identical to before.
|
||||
otherwise. The unhinted path is identical to before.
|
||||
3. The inline pass (`jolt-core/jolt/passes.clj`) propagates the hint: when it
|
||||
binds a non-trivial call argument to a fresh local, it carries the called
|
||||
function's parameter hint onto that local, so lookups inside the spliced body
|
||||
keep the bare path. Without this, inlining a hinted function would erase the
|
||||
keep the direct path. Without this, inlining a hinted function would erase the
|
||||
benefit, because the hinted parameter is replaced by an unhinted temporary.
|
||||
|
||||
The same machinery covers both `(:k m)` and `(get m :k [default])` when the key
|
||||
is a constant keyword. A `get` with a variable, numeric, or string key falls
|
||||
through to `core-get` unchanged.
|
||||
through to `jolt-get` unchanged.
|
||||
|
||||
## Record hints across namespaces, and as inference seeds
|
||||
|
||||
|
|
@ -98,15 +91,14 @@ the function where the hot reads actually happen.
|
|||
|
||||
**It resolves across namespaces.** A hint may name a record defined in another
|
||||
namespace, in either spelling — `^Vec3` where the type is `:refer`-ed, or
|
||||
`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key` in
|
||||
`src/jolt/host_iface.janet`, backed by `record-hint-ctor-key` in
|
||||
`src/jolt/evaluator.janet`) runs against the *compile* namespace and maps the
|
||||
type to its home constructor key through a constructor-value index — keyed by the
|
||||
constructor value, not a var's namespace, so a `:refer`-interned var (whose
|
||||
namespace is the referring one) still resolves home. The reader keeps a tag's
|
||||
namespace qualifier (`^v/Vec3` → `"v/Vec3"`, not `"Vec3"`) so the aliased
|
||||
spelling has something to resolve. Both `defrecord` field hints and function
|
||||
parameter hints use this resolution.
|
||||
`^v/Vec3` where the namespace is `:as`-aliased. Resolution (`record-ctor-key`,
|
||||
a `jolt.host` contract function backed by `host/chez/host-contract.ss`) runs
|
||||
against the *compile* namespace and maps the type to its home constructor key
|
||||
through a constructor-value index — keyed by the constructor value, not a var's
|
||||
namespace, so a `:refer`-interned var (whose namespace is the referring one)
|
||||
still resolves home. The reader keeps a tag's namespace qualifier (`^v/Vec3` →
|
||||
`"v/Vec3"`, not `"Vec3"`) so the aliased spelling has something to resolve. Both
|
||||
`defrecord` field hints and function parameter hints use this resolution.
|
||||
|
||||
## Soundness and the checked mode
|
||||
|
||||
|
|
@ -120,8 +112,8 @@ To make a lie visible without taxing the fast path, `JOLT_CHECK_HINTS=1` keeps
|
|||
the guard but throws on the tagged arm with a message naming the local and key:
|
||||
|
||||
```
|
||||
type hint violated on `m`: (:a m) — value carries :jolt/type
|
||||
(a phm/sorted/transient/lazy-seq), not the plain struct/record the
|
||||
type hint violated on `m`: (:a m) — value is a
|
||||
phm/sorted/transient/lazy-seq, not the plain struct/record the
|
||||
^:struct/^Record hint asserts
|
||||
```
|
||||
|
||||
|
|
@ -134,7 +126,7 @@ off). The flag is part of the image-cache fingerprint.
|
|||
Type hints parse in every position Clojure accepts them and are inert except for
|
||||
the optimization above. This matches Clojure's "parse and otherwise do nothing"
|
||||
model, with the difference that Clojure additionally uses hints to avoid
|
||||
reflection and select primitive arithmetic, which do not apply to a Janet host.
|
||||
reflection and select primitive arithmetic, which do not apply to the Chez host.
|
||||
|
||||
## Measured effect
|
||||
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ A type `T` is one of:
|
|||
`:nonnil` for "provably not nil and not false", which is what the struct-vs-phm
|
||||
decision needs; see below.)
|
||||
- `:nil`.
|
||||
- `{:struct {field -> T}}` — a raw-get-safe map (Janet struct or record) whose
|
||||
- `{:struct {field -> T}}` — a raw-get-safe map (a record) whose
|
||||
field `k` has type `(fields k)` or `:any` if absent. The degenerate
|
||||
`{:struct {}}` is "a struct, fields unknown" and replaces today's
|
||||
`:struct-map`.
|
||||
|
|
@ -65,7 +65,7 @@ A type `T` is one of:
|
|||
|
||||
Types are immutable values comparable by structural equality, exactly like the
|
||||
current `{:vec ELEM}` representation, so they flow across the portable
|
||||
inference and the Janet orchestrator unchanged.
|
||||
inference and the host unchanged.
|
||||
|
||||
### Join (least upper bound)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue