4d8494d620
A separate jolt-deps executable (mirroring jpm beside janet) resolves a deps.edn into source roots and runs jolt with them on JOLT_PATH; the runtime stays deps-agnostic. - src/jolt/deps.janet: read deps.edn (Janet parses it directly), resolve :git/* and :local/root via jpm/pm's download-bundle (git fetch + cache, no build), recurse for transitive deps, return de-duped roots. jpm loaded lazily so it's never embedded. Roots cached on a deps.edn hash. - src/jolt/deps_cli.janet + project.janet: the jolt-deps tool — path/run/repl/-e. - main: apply JOLT_PATH at runtime (the CLI ctx is frozen into the image at build, so init's env read wouldn't see it). Verified end to end against a local dep and a real git dep (medley). git only, pure clj/cljc. Test: deps-resolve-test (local deps, no network).
140 lines
6.9 KiB
Markdown
140 lines
6.9 KiB
Markdown
# Loading Clojure libraries via deps.edn (git deps, on jpm)
|
|
|
|
Research notes for letting a Jolt project pull pure-Clojure libraries through a
|
|
`deps.edn` and `(require ...)` them. Scope is deliberately narrow:
|
|
|
|
- **git deps only** — no Maven/`~/.m2` resolution.
|
|
- **pure `clj`/`cljc`** — anything needing the JVM (Java interop, host classes)
|
|
won't load or run, and that's expected.
|
|
- **no classpath machinery** — we just need `require` to find a dep's namespaces
|
|
at dev time (REPL) so they can be compiled into the image at build time.
|
|
- **piggyback on jpm** — reuse jpm's existing git fetch + cache; don't write a
|
|
package manager.
|
|
|
|
Nothing here is implemented yet. Everything below was verified against the
|
|
installed jpm and a real lib (medley).
|
|
|
|
## How jpm handles dependencies today
|
|
|
|
jpm's package code lives in `/opt/homebrew/lib/janet/jpm/pm.janet`. The relevant
|
|
pieces:
|
|
|
|
- **`resolve-bundle`** normalizes a dep spec to `{:url :tag :type :shallow}`. It
|
|
accepts a table (`:url`/`:repo`, and `:tag`/`:sha`/`:commit`/`:ref`) or a
|
|
`"url::type::tag"` string. So a deps.edn `{:git/url … :git/sha …}` maps onto it
|
|
directly.
|
|
- **`download-bundle url :git tag shallow`** clones into a cache dir and returns
|
|
the path. The cache is `(find-cache)` = `<modpath>/.cache`, and the per-dep
|
|
directory is a deterministic id: `git_<tag>_<sanitized-url>`. Under the hood
|
|
`download-git-bundle` does `git init` + `remote add origin` + fetch + reset to
|
|
the tag/sha (plus submodules). Pure git — **no build step**.
|
|
- **`bundle-install`** is the part we *don't* want: after downloading it does
|
|
`(require-jpm "./project.janet")` and runs build/install rules. A Clojure lib
|
|
has no `project.janet`, so this would fail. The clone/cache half
|
|
(`download-bundle`) is cleanly separable from this build half.
|
|
|
|
So jpm already gives us git resolution + a content-addressed cache for free; we
|
|
just skip its build phase.
|
|
|
|
### Verified
|
|
|
|
```janet
|
|
(import jpm/config :as cfg)
|
|
(import jpm/pm :as pm)
|
|
(cfg/load-default) ; sets gitpath, cache defaults, etc.
|
|
(setdyn :modpath "<project>/jpm_tree") ; where the cache lives
|
|
(def s (pm/resolve-bundle {:url "https://github.com/weavejester/medley"
|
|
:tag "1.0.0" :shallow true}))
|
|
(pm/download-bundle (s :url) (s :type) (s :tag) (s :shallow))
|
|
;; => "<modpath>/.cache/git_1.0.0_https___github.com_weavejester_medley"
|
|
;; with source at <that>/src/medley/core.cljc
|
|
```
|
|
|
|
`cfg/load-default` is required — calling `download-bundle` without jpm's config
|
|
dyns set fails in its `shell` helper.
|
|
|
|
And the source loads and runs in Jolt: `(medley.core/abs -5)` → `5`,
|
|
`(medley.core/find-first odd? [2 4 5])` → `5` (coverage is per-function; a few
|
|
hit interpreter gaps, which is fine).
|
|
|
|
## The shape
|
|
|
|
1. **Resolve.** Read `deps.edn`, take the `:deps` whose specs are git
|
|
(`:git/url` + `:git/sha`/`:git/tag`). For each, `resolve-bundle` +
|
|
`download-bundle` into the project's `jpm_tree/.cache`. Read each cloned
|
|
dep's own `deps.edn` and recurse for transitive git deps. (`:local/root`
|
|
deps are even simpler — just a path, no fetch.)
|
|
2. **Collect source roots.** A dep's source dirs are its deps.edn `:paths`
|
|
(default `["src"]`), so a root is `<clone-dir>/<path>`. The result is just an
|
|
ordered list of directories — not a classpath abstraction, a list of roots.
|
|
3. **Teach the loader the roots.** `evaluator.janet/ns->path` currently hardcodes
|
|
`src/jolt/<ns>.clj`. Generalize `maybe-require-ns` to, after the stdlib path,
|
|
search each dep root for `<ns>.clj` then `<ns>.cljc`. `src/jolt/` stays first
|
|
so the stdlib always wins. (Two small changes: a root list in the ctx, and
|
|
trying `.cljc`.)
|
|
4. **Dev vs build.**
|
|
- *Dev:* `jolt-deps` resolves `deps.edn` (cached on a hash of it, so it's a
|
|
no-op when unchanged) and runs jolt with the roots on `JOLT_PATH`;
|
|
`(require '[medley.core])` then just works.
|
|
- *Build:* the dep namespaces a project actually uses get compiled into the
|
|
image the same way the embedded `jolt.nrepl` source is today — load them at
|
|
build time so the shipped binary needs neither the deps nor jpm.
|
|
|
|
## Why this fits "no classpath"
|
|
|
|
We never construct a Java-style classpath or deal with jar extraction, ordering
|
|
semantics, or version conflict resolution. Resolution is a tree walk over git
|
|
deps; "the classpath" is just the list of `src` dirs of the clones. The loader
|
|
already does path-based namespace lookup — we widen it from one root to a few.
|
|
|
|
## A separate tool, like jpm beside janet
|
|
|
|
The jolt *runtime* knows nothing about deps.edn — exactly as the `janet` binary
|
|
knows nothing about jpm. Resolution lives in a separate `jolt-deps` tool (its own
|
|
`declare-executable`), and the runtime's only interface is the source roots in
|
|
`JOLT_PATH`:
|
|
|
|
```
|
|
jolt-deps path # print the resolved roots, ':'-joined
|
|
jolt-deps run FILE [args] # resolve, then run `jolt FILE …` with JOLT_PATH set
|
|
jolt-deps repl # resolve, then start a jolt REPL with JOLT_PATH set
|
|
jolt-deps -e EXPR [args] # resolve, then `jolt -e …`
|
|
```
|
|
|
|
It **reuses `jpm/pm`** (`resolve-bundle` + `download-bundle`, after
|
|
`jpm/config/load-default`) for the git fetch/cache rather than shelling `git` —
|
|
less code, shares jpm's cache. jpm is loaded lazily (`require`, not `import`), so
|
|
it's pulled in only when resolving (dev time), never embedded in a binary. The
|
|
internal-API risk is acceptable since it's dev-time only.
|
|
|
|
One gotcha worth recording: `jolt`'s context is built into its image at build
|
|
time, so `JOLT_PATH` is read at runtime in `main` (not in `init`, whose env read
|
|
would be frozen at build).
|
|
|
|
## Limitations
|
|
|
|
- Pure `clj`/`cljc` only. JVM interop, host classes, and unimplemented
|
|
`clojure.core` corners fail — expected, not a goal.
|
|
- Per-function coverage: a namespace can load with most functions working and a
|
|
few not.
|
|
- Source only; compiled `.class` files (if any in a git dep) are ignored.
|
|
|
|
## Plan
|
|
|
|
1. **Loader roots.** *(done)* `maybe-require-ns` now searches an ordered root
|
|
list (`ctx.env :source-paths`, stdlib first) trying `.clj` then `.cljc`;
|
|
`init` adds roots from the `:paths` opt and `JOLT_PATH` (colon-separated).
|
|
Verified loading a real lib (medley) from an added root. See
|
|
`test/integration/deps-loader-test.janet`.
|
|
2. **Resolve git deps via jpm.** *(done)* `src/jolt/deps.janet` reads `deps.edn`,
|
|
resolves `:git/*` + `:local/root` through `jpm/pm` into `jpm_tree/.cache`,
|
|
recurses for transitive deps, and returns the roots (cached on a deps.edn
|
|
hash). The separate `jolt-deps` tool (`src/jolt/deps_cli.janet`,
|
|
`declare-executable`) exposes `path`/`run`/`repl`/`-e`. See
|
|
`test/integration/deps-resolve-test.janet`.
|
|
3. **Build-time compile-in.** Fold the used dep namespaces into the image at
|
|
build (as with embedded `jolt.nrepl`), so a built artifact needs neither the
|
|
deps nor jpm. *(not started)*
|
|
4. **Conformance.** Pull a few popular pure-`cljc` git libs, see what loads/runs,
|
|
and drive interpreter gaps from the failures — same loop as the
|
|
clojure-test-suite battery.
|