From 5e20ff21ff0c461367f9891eda5b05ac4092b65a Mon Sep 17 00:00:00 2001 From: Yogthos Date: Fri, 5 Jun 2026 22:28:22 -0400 Subject: [PATCH] Research notes: loading Clojure libs via deps.edn Findings and a phased plan for consuming deps.edn so projects can require real Clojure libraries. Jolt already reads EDN, jars ship .clj/.cljc source, and a real lib (medley) loads and runs. The gaps are a single-rooted loader (needs a classpath + .cljc) and dependency resolution. Recommends reusing 'clojure -Spath' to start, then native git-dep resolution; jpm stays the Janet build tool alongside this. See doc/tools-deps.md. --- doc/tools-deps.md | 154 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 doc/tools-deps.md diff --git a/doc/tools-deps.md b/doc/tools-deps.md new file mode 100644 index 0000000..08a26b0 --- /dev/null +++ b/doc/tools-deps.md @@ -0,0 +1,154 @@ +# Loading Clojure libraries via deps.edn + +Research notes on letting Jolt consume `deps.edn` so a project can pull real +Clojure libraries and `(require ...)` them. This documents what works today, what +it would take, and a recommended path. Nothing here is implemented yet. + +## Goal + +Given a `deps.edn` like + +```clojure +{:paths ["src"] + :deps {medley/medley {:mvn/version "1.0.0"} + some/gitlib {:git/url "https://..." :git/sha "..."}}} +``` + +run `jolt` in that directory and have `(require '[medley.core :as m])` find and +load the library's source from the resolved dependency, the same way the stdlib +is loaded today. + +## What works today + +- **Jolt reads EDN.** `(read-string (slurp "deps.edn"))` parses a deps.edn into a + Jolt map — no extra parser needed. +- **Library source ships in the jars.** Maven Clojure jars contain the `.clj` / + `.cljc` source at namespace-matching paths (e.g. `medley/core.cljc`, + `msgpack/core.clj`), not just compiled `.class` files. So we never need a JVM + to *run* the code — only to fetch/resolve it, and even that is optional (below). +- **Jolt can run real library source.** Loading `medley/core.cljc` straight from + its jar works: `(medley.core/abs -5)` → `5`, `(medley.core/find-first odd? …)` + → `5`. Some functions hit features Jolt doesn't fully support yet — coverage is + per-function, not all-or-nothing. +- **The real resolver is on the box.** `clojure`, `clj`, and `mvn` are installed, + with `~/.m2` and `~/.gitlibs` populated. `clojure -Spath` already prints the + fully-resolved, transitive classpath (dirs + jars). + +## What's missing + +The loader is single-rooted. `evaluator.janet/ns->path` hardcodes: + +```janet +(string "src/jolt/" (dots->slashes (dashes->underscores ns)) ".clj") +``` + +and `maybe-require-ns` loads exactly that one path if it exists. To load deps we +need: + +1. **A classpath** — a list of source roots searched in order, not one fixed + prefix. Roots = `:paths` from deps.edn + each resolved dependency's source. +2. **`.cljc` support** — try `foo/bar.cljc` as well as `foo/bar.clj` (most libs + ship `.cljc` or `.clj`; the loader only tries `.clj` today). +3. **`ns`-form handling on load.** Stdlib files have no `ns` form, so + `maybe-require-ns` sets the current ns manually before loading. Library files + *do* have `(ns ...)`. Both already work in practice (the `ns` form re-asserts + the namespace), but the loader should not assume "no ns form." + +## Resolving dependencies — three options + +The hard part is turning coordinates into local source roots. Maven resolution +(transitive deps, version conflict resolution, POM parsing) is real work; git +deps are comparatively easy. + +### A. Shell out to the Clojure CLI (recommended first cut) + +Run `clojure -Spath` (optionally `-Sdeps`/aliases) in the project dir, capture +the `:`-separated classpath, then: + +- directory entries → add directly as source roots; +- jar entries → extract `*.clj` / `*.cljc` into a cache dir + (`.jolt/classpath//`) with `unzip`/`jar` (both present; Janet has no + built-in zip) and add the cache dir as a root. + +Pros: reuses the canonical resolver, so transitive deps, exclusions, aliases, and +version conflict resolution are all correct and match what JVM Clojure sees. Tiny +amount of code. + +Cons: requires the Clojure CLI (hence a JVM) *at resolve time*. Runtime stays +JVM-free. We'd cache the result so resolution only reruns when `deps.edn` +changes. + +### B. Jolt-native resolver + +Parse `deps.edn` ourselves and resolve: + +- `:git/url` + `:git/sha` → `git clone`/checkout into a cache (this is roughly + what jpm already does for Janet git deps — see "jpm" below); +- `:mvn/version` → download the POM + jar from Maven Central over HTTP, parse the + POM for transitive deps, resolve versions. + +Pros: no JVM dependency at all; self-contained. + +Cons: reimplementing Maven resolution (POM transitive graph, `:exclusions`, +nearest-wins version selection) is the bulk of tools.deps and easy to get subtly +wrong. Large effort. + +### C. Hybrid + +Native path for `:paths` and `:git/*` deps (cheap, no JVM); shell to `clojure +-Spath` only when `:mvn/*` deps are present. Gives a JVM-free experience for +git-only / local projects and correct Maven resolution when needed. + +## Where jpm fits + +jpm builds the Jolt binary and manages *Janet* packages; it has no Maven/Clojure +notion, so deps.edn support sits beside jpm rather than inside it. Two useful +touch points: + +- jpm already fetches and caches **git** repositories for Janet deps — the same + machinery (or `~/.gitlibs`) can back option B/C's git-dep handling, so we don't + write a git cache from scratch. +- A project-level `jpm` rule (in `project.janet`) could run resolution as a build + step and write a classpath file, for projects that want deps resolved at build + time rather than first run. + +But the primary integration is at the Jolt runtime/CLI, not jpm: see below. + +## Proposed shape + +- **Classpath in the context.** Add a `:classpath` (ordered list of roots) to the + ctx env. `ns->path` becomes "search each root for `foo/bar.clj` then + `foo/bar.cljc`", with `src/jolt/` always first so the stdlib wins. +- **Resolution step.** On startup (or via `jolt deps`), if `deps.edn` exists, + resolve it to roots (option A to start) and set `:classpath`. Cache keyed on a + hash of `deps.edn` so it's a no-op when unchanged. +- **Config knobs.** `JOLT_CLASSPATH` env / `--classpath` flag to set roots + directly (bypassing resolution), mirroring how `JOLT_MUTABLE` works. + +## Limitations (set expectations) + +- **JVM-only libraries don't run.** Anything depending on Java interop, host + classes, or `clojure.core` features Jolt lacks will fail to load or fail at a + call. Target audience is pure-`clj`/`cljc` libraries. +- **Coverage is per-function.** As the medley probe showed, a namespace can load + and have most functions work while a few hit unimplemented core behavior. +- **No AOT/`.class` execution** — ever. We only consume source from the + classpath; compiled classes in jars are ignored. +- Macro/protocol/reader-conditional support is whatever the Jolt interpreter + already provides (reader conditionals `#?` are supported, which is why `.cljc` + loads). + +## Recommended plan (phased) + +1. **Loader classpath.** Generalize `ns->path`/`maybe-require-ns` to search an + ordered root list and try `.clj` + `.cljc`. Add `JOLT_CLASSPATH`/`--classpath`. + No resolution yet — point it at a directory of source by hand and load a lib. + (Unblocks everything; independently testable.) +2. **deps.edn → classpath via `clojure -Spath` (option A).** Resolve, extract + jar source to a cache, set the classpath. `jolt deps` to resolve/print; + auto-resolve on startup when `deps.edn` is present. +3. **Native git deps (toward option C).** Resolve `:git/*` (and `:local/root`) + without the JVM, falling back to the CLI only for `:mvn/*`. +4. **Conformance pass.** Pull a handful of popular pure-`cljc` libs, see what + loads/runs, and use the failures to drive interpreter gaps — same loop as the + clojure-test-suite battery.