Source locations: reader positions, error locations, native stack traces (#218)

* Reader records source line/column on list forms

The reader stamps 1-based :line/:column metadata on every list form (plus
:file when load-jolt-file is reading a file), and jolt.host/form-position
reads it back so the analyzer's :pos scaffold finally gets real data. A
left-to-right cursor counts newlines over the delta between successive forms,
so it stays O(n). Vector/map/set literals are untouched (their metadata is a
runtime value the analyzer would have to wrap in with-meta); empty () can't
carry meta. ^meta now merges onto the position keys instead of clobbering them.

Re-mint is byte-identical (the backend doesn't emit :pos), so this is a pure
scaffold for the error-location work that follows.

* Report source location on uncaught errors

Each top-level form records its source position (thread-local) before it
compiles+evals, and cli.ss jolt-report-uncaught appends 'at file:line:col'
when an error propagates out. Covers joltc -e, joltc run <file>, and
load-string — every interpreted path. Top-level granularity, one set per
form; deeper frames come from the Phase 2 frame walk.

Runtime .ss only, no re-mint.

* Clojure stack traces via source registry + native frame walk

A direct-link build emits (jolt-register-source! short-name ns name file line)
once per fn def — at definition time, so zero per-call cost. On an uncaught
error the reporter walks Chez's native continuation frames (jolt-throw captures
the live continuation via call/cc; host conditions carry their own
&continuation), maps each frame's procedure name through the registry, and
prints a Clojure backtrace 'ns/name (file:line)'. Wired into both the cli and a
built binary's launcher.

Frames are keyed by the short munged fn name Chez actually reports (emit-fn's
letrec self-binding), not jv$ns$name; a cross-namespace collision degrades to
the bare frame name rather than a wrong attribution. The analyzer carries the
original form's position through defn macroexpansion onto the def node.

Calling a non-fn now throws a catchable ClassCastException (via jolt-throw)
naming the operator, instead of a raw Chez error.

Caveats (documented in source-registry.ss): names map only in direct-link/AOT
closed-world builds — the open-world -e/repl/run path falls back to the
top-level location; and pervasive TCO erases tail-call frames, so a mapped
trace shows only the non-tail spine. JOLT_DEBUG_FRAMES dumps raw frame names.

Re-mint (analyzer + backend); prelude byte-identical (direct-link off during
mint). Corpus rows certified, build-smoke asserts the trace.

* Propagate source position through macroexpansion

hc-expand-1 now carries the macro call form's :line/:column onto the top of a
list expansion that has none of its own (merged under any meta the macro set),
so errors and stack traces in macro-generated code point at the call site —
Clojure parity. The analyze recursion re-expands inner macros, so each level's
top form picks it up, matching the reference compiler. (meta (macroexpand-1
'(when x y))) now reports the call-site line.

A direct-link fn defined through a user macro (build-app's defguarded) registers
with a real line, so build-smoke's trace assertion covers macro-defined fns.

Runtime .ss (host-contract.ss) — no re-mint; selfhost holds.

Phase 3's optional items are deferred: :line-in-ex-data has no clean consumer
(it would pollute ex-data, break = and printing, and positions already surface
via the trace + top-level location), and Chez source-object emission is a large
backend change the jv$-name registry already sidesteps.

* Review fixes: registration key, thread-locals, debug flag timing

- Register a fn under the name Chez actually reports for its frame, not the def
  name: a named fn literal whose name differs from the def (def foo (fn bar …))
  is framed as 'bar', and an anonymous fn def (def foo (fn …)) as jv$ns$foo.
  Both previously registered under the def name and so never appeared in traces.
- rdr-source-file / rdr-pos-cursor are thread parameters, so concurrent compiles
  (futures, core.async) don't clobber each other's file/line attribution.
- Read JOLT_DEBUG_FRAMES at call time: a built binary evaluates top-level forms
  at heap-build time, where a load-time getenv is always unset.

Re-mint (backend + reader); prelude byte-identical, selfhost holds.

---------

Co-authored-by: Yogthos <yogthos@gmail.com>
This commit is contained in:
Dmitri Sotnikov 2026-06-26 02:14:34 +00:00 committed by GitHub
parent bdf436e242
commit 8180c85393
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 449 additions and 122 deletions

View file

@ -316,7 +316,11 @@
;; (with-meta form meta) for a meta-carrying collection literal in code, so
;; (meta ^{:tag :int} [1 2]) / ^:foo {} still works.
(else
(let ((c (jolt-with-meta target meta)))
;; Merge onto any metadata the target already carries (a list form picks up
;; :line/:column first, then ^meta folds its keys on top).
(let* ((old (jolt-meta target))
(merged (rdr-merge-meta (if (jolt-nil? old) jolt-nil old) meta))
(c (jolt-with-meta target merged)))
;; jolt-with-meta copies a pmap, giving it a fresh identity the rdr-map-order
;; side-table (source key order for left-to-right map-literal eval) loses —
;; carry the order entry over to the copy.
@ -324,6 +328,45 @@
(when order (hashtable-set! rdr-map-order c order)))
c))))
;; --- source position --------------------------------------------------------
;; List forms (code) carry 1-based :line/:column, plus :file when the compiler
;; bound rdr-source-file. read-string leaves the file unset. The analyzer reads
;; this back via jolt.host/form-position to stamp :pos on call nodes; macros and
;; (meta (read-string "(…)")) see it too.
(define rdr-source-file (make-thread-parameter #f))
(define rdr-kw-line (keyword #f "line"))
(define rdr-kw-column (keyword #f "column"))
(define rdr-kw-file (keyword #f "file"))
;; Forms are read left-to-right, so the indices queried are non-decreasing within
;; one source string — keep a cursor and count newlines only over the delta
;; (O(n) total, not O(n^2)). A different string or a backward index resets it.
(define rdr-pos-cursor (make-thread-parameter #f)) ; #f | (vector s i line col)
(define (rdr-line-col-at s i)
(let* ((cur (rdr-pos-cursor))
(reuse (and (vector? cur) (eq? (vector-ref cur 0) s)
(fx<=? (vector-ref cur 1) i)))
(k0 (if reuse (vector-ref cur 1) 0))
(l0 (if reuse (vector-ref cur 2) 1))
(c0 (if reuse (vector-ref cur 3) 1)))
(let loop ((k k0) (line l0) (col c0))
(if (fx>=? k i)
(begin (rdr-pos-cursor (vector s k line col)) (values line col))
(if (char=? (string-ref s k) #\newline)
(loop (fx+ k 1) (fx+ line 1) 1)
(loop (fx+ k 1) line (fx+ col 1)))))))
(define (rdr-pos-meta line col)
(let ((f (rdr-source-file)))
(if f
(jolt-hash-map rdr-kw-line line rdr-kw-column col rdr-kw-file f)
(jolt-hash-map rdr-kw-line line rdr-kw-column col))))
(define (rdr-attach-pos lst line col)
(if (empty-list-t? lst) ; () is interned, can't carry meta (= Clojure)
lst
(rdr-attach-meta lst (rdr-pos-meta line col))))
;; --- # dispatch -------------------------------------------------------------
;; #(...) anonymous fn shorthand: % -> p1, %N -> pN, %& -> rest. The
;; fixed arity is the MAX positional used (Clojure: #(do %2 %&) -> [p1 p2 & rest]).
@ -496,8 +539,9 @@
(values rdr-eof i)
(let ((c (string-ref s i)))
(cond
((char=? c #\() (let-values (((es j) (rdr-read-seq s (+ i 1) end #\))))
(values (apply jolt-list es) j)))
((char=? c #\() (let-values (((line col) (rdr-line-col-at s i)))
(let-values (((es j) (rdr-read-seq s (+ i 1) end #\))))
(values (rdr-attach-pos (apply jolt-list es) line col) j))))
((char=? c #\[) (let-values (((es j) (rdr-read-seq s (+ i 1) end #\])))
(values (apply jolt-vector es) j)))
((char=? c #\{) (let-values (((es j) (rdr-read-seq s (+ i 1) end #\})))